Policy Improvement Directions for Reinforcement Learning in Reproducing Kernel Hilbert Spaces

Paternain, Santiago; Bazerque, Juan Andrés; Small, Austin; Ribeiro, Alejandro

doi:10.1109/cdc40024.2019.9029198

Cited by 3 publications

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Building on our preliminary results [21], we establish in Theorem 1 that the gradients of the value function at any state are also ascent directions of the value function at the initial state. Leveraging this result we address the convergence of the online policy gradient algorithm to a neighborhood of the critical points in Theorem 2, hence dropping the assumption of the convergence to-and existence of-the stationary distribution over states for every intermediate policy.…”

Section: Introductionmentioning

confidence: 81%

Policy Gradient for Continuing Tasks in Non-stationary Markov Decision Processes

Paternain¹,

Bazerque²,

Ribeiro³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning considers the problem of finding policies that maximize an expected cumulative reward in a Markov decision process with unknown transition probabilities. In this paper we consider the problem of finding optimal policies assuming that they belong to a reproducing kernel Hilbert space (RKHS). To that end we compute unbiased stochastic gradients of the value function which we use as ascent directions to update the policy. A major drawback of policy gradient-type algorithms is that they are limited to episodic tasks unless stationarity assumptions are imposed. Hence preventing these algorithms to be fully implemented online, which is a desirable property for systems that need to adapt to new tasks and/or environments in deployment. The main requirement for a policy gradient algorithm to work is that the estimate of the gradient at any point in time is an ascent direction for the initial value function. In this work we establish that indeed this is the case which enables to show the convergence of the online algorithm to the critical points of the initial value function. A numerical example shows the ability of our online algorithm to learn to solve a navigation and surveillance problem, in which an agent must loop between to goal locations. This example corroborates our theoretical findings about the ascent directions of subsequent stochastic gradients. It also shows how the agent running our online algorithm succeeds in learning to navigate, following a continuing cyclic trajectory that does not comply with the standard stationarity assumptions in the literature for non episodic training.

show abstract

Section: Introductionmentioning

confidence: 81%