Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert Spaces

Abstract

Reinforcement learning consists of finding policies that maximize an expected cumulative long term reward in a Markov decision process with unknown transition probabilities and instantaneous rewards. In this paper we consider the problem of finding such optimal policies while assuming they are continuous functions belonging to a reproducing kernel Hilbert space (RKHS). To learn the optimal policy we introduce a stochastic policy gradient ascent algorithm with three unique novel features: (i) The stochastic estimates of policy gradients are unbiased. (ii) The variance of stochastic gradients is reduced drawing on ideas from numerical differentiation. (iii) Policy complexity is controlled using sparse RKHS representations. Novel feature (i) is instrumental in proving convergence to a stationary point of the expected cumulative reward. Novel feature (ii) facilitates reasonable convergence times. Novel feature (iii) is a necessity in practical implementations which we show can be done in a way that does not eliminate convergence guarantees. Numerical examples in standard problems illustrate successful learning of policies with low complexity representations which are close to stationary points of the expected cumulative reward.

Safe Policies for Reinforcement Learning via Primal-Dual Methods

“…In this remark we discuss the equivalence between the formulations in (1) and (2). This discussion is inspired in [5, Section 2.3] and in the proofs of [32,Proposition 2 and 3]. Let us start by considering the finite horizon value function in (1) with a horizon chosen from a geometric distribution with parameter γ ∈ (0, 1).…”

“…Under mild assumptions on the reward function it is possible to exchange the sum and the expectation (see e.g., [32,Proposition 2] ). Also assuming that the horizon is drawn independently from the trajectory, we can write…”

“…where in the case of the discounted problem T is allowed to take the value infinity and γ ∈ (0, 1) and for the episodic task γ = 1. Problem ( 23) can be therefore be solved with classic reinforcement learning algorithms such as policy gradient [8], [9], [32] or actor-critic methods [37]. Notice that the reward r λ (s, a) is similar to that defined in (4).…”

Towards Safe Continuing Task Reinforcement Learning

“…Overall, the resulting procedure takes the form shown in Algorithm 1. At time k, the agent advances the system by T ∼ Geom(γ) steps and thus obtains a sample s k from the occupancy measure µ s k [26]. Similarly, in order to obtain estimates of ∇V λ k s k (θ k ) and U s k (θ k ), the agent can do so by accumulating the rewards r(s t , a t ) and constraint satisfaction 1(s ∈ S safe ) by performing another rollout of T Q ∼ Geom(γ) time steps.…”

“…Since both series are convergent (by hypothesis, otherwise ρ z (s) would not be a proper probability measure), the summations in (26) can be rearranged to get…”