Policy gradients with memory-augmented critic: Stabilizing off-policy policy gradients via differentiable memory

Takuma Seno, Michita Imai

Research output: Contribution to journalArticlepeer-review

Abstract

Deep reinforcement learning has been investigated in high-dimensional continuous control tasks. Deep Deterministic Policy Gradients (DDPG) is known as a highly sample-efficient policy gradients algorithm. However, it is reported that DDPG is unstable during training due to bias and variance problems of learning its action-value function. In this paper, we propose Policy Gradients with Memory Augmented Critic (PGMAC) that builds action-value function with the memory module previously proposed as Differentiable Neural Dictionary (DND). Although the DND is only studied in discrete action-space problems, we propose Action-Concatenated Key, which is a technique to combine DDPG-based policy gradient methods and DND. Furthermore, the remarkable advantage of PGMAC is shown that long-term reward calculation and weighted summation of value estimation at DND has an essential mechanism to solve the bias and variance problem. In experiment, PGMAC significantly outperformed baselines in continuous control tasks. The effects of hyperparameters were also investigated to show that the memory-augmented action-value function reduces the bias and variance in policy optimization.

Original languageEnglish
Article numberB-K71_1-8
Pages (from-to)1-8
Number of pages8
JournalTransactions of the Japanese Society for Artificial Intelligence
Volume36
Issue number1
DOIs
Publication statusPublished - 2021

Keywords

  • Continuous control
  • Deep reinforcement learning
  • Memory module
  • Policy gradients

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Policy gradients with memory-augmented critic: Stabilizing off-policy policy gradients via differentiable memory'. Together they form a unique fingerprint.

Cite this