AN FPGA-BASED ON-DEVICE REINFORCEMENT LEARNING APPROACH USING ONLINE SEQUENTIAL LEARNING

Hirohisa Watanabe, Mineto Tsukada, Hiroki Matsutani

Research output: Contribution to journalArticlepeer-review

Abstract

DQN (Deep Q-Network) is a method to perform Q-learning for reinforcement learning using deep neural networks. DQNs require large buffers for experience reply and rely on backpropagation based iterative optimization, making them difficult to be implemented on resource-limited edge devices. In this paper, we propose a lightweight on-device reinforcement learning approach for low-cost FPGA devices. It exploits a recently proposed neural-network based on-device learning approach that does not rely on the backpropagation method but uses ELM (Extreme Learning Machine) and OS-ELM (Online Sequential ELM) based training algorithms. In addition, we propose a combination of L2 regularization and spectral normalization for the on-device reinforcement learning, so that output values of the neural networks can be fit into a certain range and the reinforcement learning becomes stable. The proposed reinforcement learning approach is designed for Xilinx PYNQ-Z1 board as a low-cost FPGA platform. The experiment results using OpenAI Gym demonstrate that the proposed algorithm and its FPGA implementation complete a CartPole-v0 task 29.76x and 126.06x faster than a conventional DQN-based approach when the number of hidden-layer nodes is 64.

Original languageEnglish
JournalUnknown Journal
Publication statusPublished - 2020 May 10

Keywords

  • FPGA
  • On-device learning
  • OS-ELM
  • Reinforcement learning
  • Spectral normalization

ASJC Scopus subject areas

  • General

Fingerprint Dive into the research topics of 'AN FPGA-BASED ON-DEVICE REINFORCEMENT LEARNING APPROACH USING ONLINE SEQUENTIAL LEARNING'. Together they form a unique fingerprint.

Cite this