An FPGA-Based On-Device Reinforcement Learning Approach using Online Sequential Learning

Hirohisa Watanabe, Mineto Tsukada, Hiroki Matsutani

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

DQN (Deep Q-Network) is a method to perform Q-learning for reinforcement learning using deep neural networks. DQNS require a large buffer and batch processing for an experience replay and rely on a backpropagation based iterative optimization, making them difficult to be implemented on resource-limited edge devices. In this paper, we propose a lightweight on-device reinforcement learning approach for low-cost FPGA devices. It exploits a recently proposed neural-network based on-device learning approach that does not rely on the backpropagation method but uses OS-ELM (Online Sequential Extreme Learning Machine) based training algorithm. In addition, we propose a combination of L2 regularization and spectral normalization for the on-device reinforcement learning so that output values of the neural network can be fit into a certain range and the reinforcement learning becomes stable. The proposed reinforcement learning approach is designed for PYNQ-Z1 board as a low-cost FPGA platform. The evaluation results using OpenAI Gym demonstrate that the proposed algorithm and its FPGA implementation complete a CartPole-v0 task 29.77x and 89.40x faster than a conventional DQN-based approach when the number of hidden-layer nodes is 64.

Original languageEnglish
Title of host publication2021 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2021 - In conjunction with IEEE IPDPS 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages96-103
Number of pages8
ISBN (Electronic)9781665435772
DOIs
Publication statusPublished - 2021 Jun
Event2021 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2021 - Virtual, Portland, United States
Duration: 2021 May 17 → …

Publication series

Name2021 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2021 - In conjunction with IEEE IPDPS 2021

Conference

Conference2021 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2021
Country/TerritoryUnited States
CityVirtual, Portland
Period21/5/17 → …

Keywords

  • FPGA
  • On device learning
  • OS ELM
  • Reinforcement learning
  • Spectral normalization

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems

Fingerprint

Dive into the research topics of 'An FPGA-Based On-Device Reinforcement Learning Approach using Online Sequential Learning'. Together they form a unique fingerprint.

Cite this