Policy Gradient for Continuing Tasks in Non-stationary Markov Decision Processes

Abstract

Reinforcement learning considers the problem of finding policies that maximize an expected cumulative reward in a Markov decision process with unknown transition probabilities. In this paper we consider the problem of finding optimal policies assuming that they belong to a reproducing kernel Hilbert space (RKHS). To that end we compute unbiased stochastic gradients of the value function which we use as ascent directions to update the policy. A major drawback of policy gradient-type algorithms is that they are limited to episodic tasks unless stationarity assumptions are imposed. Hence preventing these algorithms to be fully implemented online, which is a desirable property for systems that need to adapt to new tasks and/or environments in deployment. The main requirement for a policy gradient algorithm to work is that the estimate of the gradient at any point in time is an ascent direction for the initial value function. In this work we establish that indeed this is the case which enables to show the convergence of the online algorithm to the critical points of the initial value function. A numerical example shows the ability of our online algorithm to learn to solve a navigation and surveillance problem, in which an agent must loop between to goal locations. This example corroborates our theoretical findings about the ascent directions of subsequent stochastic gradients. It also shows how the agent running our online algorithm succeeds in learning to navigate, following a continuing cyclic trajectory that does not comply with the standard stationarity assumptions in the literature for non episodic training.

Towards Safe Continuing Task Reinforcement Learning

“…While Theorem 1 provides an explicit bound on the admissible difference between occupation measures starting from different states ∆ s0,s k TV , it does not inform when these occupation measures are similar. Using a different argument, [28] shows that the inner products in ( 13) can be made positive by restricting the policy parametrization. In contrast, using the occupation measure argument from Theorem 1, we provide conditions of the underlying dynamical system and control problem (discount factor γ) that are independent of the parametrization used for the policy.…”