Regularized Fitted Q-Iteration: Application to Planning

Abstract

Abstract. We consider planning in a Markovian decision problem, i.e., the problem of finding a good policy given access to a generative model of the environment. We propose to use fitted Q-iteration with penalized (or regularized) least-squares regression as the regression subroutine to address the problem of controlling model-complexity. The algorithm is presented in detail for the case when the function space is a reproducingkernel Hilbert space underlying a user-chosen kernel function. We derive bounds on the quality of the solution and argue that data-dependent penalties can lead to almost optimal performance. A simple example is used to illustrate the benefits of using a penalized procedure.

Algorithms for Reinforcement Learning

“…Although the approaches above are inspired by principled methods of supervised learning, not much is known about their statistical properties. Recently, Farahmand et al (2009Farahmand et al ( , 2008 have developed another regularization-based approach that comes with statistical guarantees. The difficulty of using (some) nonparametric techniques is that they are computationally expensive.…”

Model-based and model-free reinforcement learning for visual servoing

“…This way we hope to bring the strength of a powerful supervised learning algorithm to the planning problem. See [12] for more information about RFQI and more precise statements about its theoretical guarantees. It is noteworthy to mention that there have been a few attempts to use regularization in reinforcement learning such as [21] and [22].…”

“…We refer the reader to [23] and [12] for further details. Reader who is not interested in rigorous definitions or is already familiar with them may just skip to Section IV-B.…”

“…This globally-valid model can be used for designing a local controller and/or global planner. Afterwards, Regularized Fitted Q-Iteration (RFQI) [12] will be introduced that finds a close to optimal solution for the learning/planning problem (Section IV). This recently proposed nonparametric reinforcement learning (RL) method uses joint values data and a reward signal to find a policy that maximizes an objective functional.…”

Approximate model-assisted Neural Fitted Q-Iteration

“…For instance, Farahmand et al [3] have used regularized least squares for regularized fitted Q-learning. When using neural networks, weight decay can likewise be used.…”