Policy gradients with memory-augmented critic: Stabilizing off-policy policy gradients via differentiable memory

Takuma Seno, Michita Imai

研究成果: Article査読

抄録

Deep reinforcement learning has been investigated in high-dimensional continuous control tasks. Deep Deterministic Policy Gradients (DDPG) is known as a highly sample-efficient policy gradients algorithm. However, it is reported that DDPG is unstable during training due to bias and variance problems of learning its action-value function. In this paper, we propose Policy Gradients with Memory Augmented Critic (PGMAC) that builds action-value function with the memory module previously proposed as Differentiable Neural Dictionary (DND). Although the DND is only studied in discrete action-space problems, we propose Action-Concatenated Key, which is a technique to combine DDPG-based policy gradient methods and DND. Furthermore, the remarkable advantage of PGMAC is shown that long-term reward calculation and weighted summation of value estimation at DND has an essential mechanism to solve the bias and variance problem. In experiment, PGMAC significantly outperformed baselines in continuous control tasks. The effects of hyperparameters were also investigated to show that the memory-augmented action-value function reduces the bias and variance in policy optimization.

本文言語English
論文番号B-K71_1-8
ページ(範囲)1-8
ページ数8
ジャーナルTransactions of the Japanese Society for Artificial Intelligence
36
1
DOI
出版ステータスPublished - 2021

ASJC Scopus subject areas

  • ソフトウェア
  • 人工知能

フィンガープリント

「Policy gradients with memory-augmented critic: Stabilizing off-policy policy gradients via differentiable memory」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル