Continuous-time value function approximation in reproducing kernel hilbert spaces

Motoya Ohnishi, Mikael Johansson, Masahiro Yukawa, Masashi Sugiyama

Research output: Contribution to journalConference article

Abstract

Motivated by the success of reinforcement learning (RL) for discrete-time tasks such as AlphaGo and Atari games, there has been a recent surge of interest in using RL for continuous-time control of physical systems (cf. many challenging tasks in OpenAI Gym and DeepMind Control Suite). Since discretization of time is susceptible to error, it is methodologically more desirable to handle the system dynamics directly in continuous time. However, very few techniques exist for continuous-time RL and they lack flexibility in value function approximation. In this paper, we propose a novel framework for model-based continuous-time value function approximation in reproducing kernel Hilbert spaces. The resulting framework is so flexible that it can accommodate any kind of kernel-based approach, such as Gaussian processes and kernel adaptive filters, and it allows us to handle uncertainties and nonstationarity without prior knowledge about the environment or what basis functions to employ. We demonstrate the validity of the presented framework through experiments.

Original languageEnglish
Pages (from-to)2813-2824
Number of pages12
JournalAdvances in Neural Information Processing Systems
Volume2018-December
Publication statusPublished - 2018 Jan 1
Event32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Montreal, Canada
Duration: 2018 Dec 22018 Dec 8

Fingerprint

Hilbert spaces
Reinforcement learning
Adaptive filters
Dynamical systems
Experiments

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Continuous-time value function approximation in reproducing kernel hilbert spaces. / Ohnishi, Motoya; Johansson, Mikael; Yukawa, Masahiro; Sugiyama, Masashi.

In: Advances in Neural Information Processing Systems, Vol. 2018-December, 01.01.2018, p. 2813-2824.

Research output: Contribution to journalConference article

Ohnishi, Motoya ; Johansson, Mikael ; Yukawa, Masahiro ; Sugiyama, Masashi. / Continuous-time value function approximation in reproducing kernel hilbert spaces. In: Advances in Neural Information Processing Systems. 2018 ; Vol. 2018-December. pp. 2813-2824.
@article{49394716e9004447af7387e386520d65,
title = "Continuous-time value function approximation in reproducing kernel hilbert spaces",
abstract = "Motivated by the success of reinforcement learning (RL) for discrete-time tasks such as AlphaGo and Atari games, there has been a recent surge of interest in using RL for continuous-time control of physical systems (cf. many challenging tasks in OpenAI Gym and DeepMind Control Suite). Since discretization of time is susceptible to error, it is methodologically more desirable to handle the system dynamics directly in continuous time. However, very few techniques exist for continuous-time RL and they lack flexibility in value function approximation. In this paper, we propose a novel framework for model-based continuous-time value function approximation in reproducing kernel Hilbert spaces. The resulting framework is so flexible that it can accommodate any kind of kernel-based approach, such as Gaussian processes and kernel adaptive filters, and it allows us to handle uncertainties and nonstationarity without prior knowledge about the environment or what basis functions to employ. We demonstrate the validity of the presented framework through experiments.",
author = "Motoya Ohnishi and Mikael Johansson and Masahiro Yukawa and Masashi Sugiyama",
year = "2018",
month = "1",
day = "1",
language = "English",
volume = "2018-December",
pages = "2813--2824",
journal = "Advances in Neural Information Processing Systems",
issn = "1049-5258",

}

TY - JOUR

T1 - Continuous-time value function approximation in reproducing kernel hilbert spaces

AU - Ohnishi, Motoya

AU - Johansson, Mikael

AU - Yukawa, Masahiro

AU - Sugiyama, Masashi

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Motivated by the success of reinforcement learning (RL) for discrete-time tasks such as AlphaGo and Atari games, there has been a recent surge of interest in using RL for continuous-time control of physical systems (cf. many challenging tasks in OpenAI Gym and DeepMind Control Suite). Since discretization of time is susceptible to error, it is methodologically more desirable to handle the system dynamics directly in continuous time. However, very few techniques exist for continuous-time RL and they lack flexibility in value function approximation. In this paper, we propose a novel framework for model-based continuous-time value function approximation in reproducing kernel Hilbert spaces. The resulting framework is so flexible that it can accommodate any kind of kernel-based approach, such as Gaussian processes and kernel adaptive filters, and it allows us to handle uncertainties and nonstationarity without prior knowledge about the environment or what basis functions to employ. We demonstrate the validity of the presented framework through experiments.

AB - Motivated by the success of reinforcement learning (RL) for discrete-time tasks such as AlphaGo and Atari games, there has been a recent surge of interest in using RL for continuous-time control of physical systems (cf. many challenging tasks in OpenAI Gym and DeepMind Control Suite). Since discretization of time is susceptible to error, it is methodologically more desirable to handle the system dynamics directly in continuous time. However, very few techniques exist for continuous-time RL and they lack flexibility in value function approximation. In this paper, we propose a novel framework for model-based continuous-time value function approximation in reproducing kernel Hilbert spaces. The resulting framework is so flexible that it can accommodate any kind of kernel-based approach, such as Gaussian processes and kernel adaptive filters, and it allows us to handle uncertainties and nonstationarity without prior knowledge about the environment or what basis functions to employ. We demonstrate the validity of the presented framework through experiments.

UR - http://www.scopus.com/inward/record.url?scp=85064833306&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064833306&partnerID=8YFLogxK

M3 - Conference article

VL - 2018-December

SP - 2813

EP - 2824

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

SN - 1049-5258

ER -