Regularized Fitted Q-Iteration for planning in continuous-space Markovian decision problems

Abstract

Reinforcement learning with linear and non-linear function approximation has been studied extensively in the last decade. However, as opposed to other fields of machine learning such as supervised learning, the effect of finite sample has not been thoroughly addressed within the reinforcement learning framework. In this paper we propose to use regularization in reinforcement learning and planning. More specifically, we control the complexity of the value function approximation using L 2 regularization. We consider the fitted Q-iteration algorithm, provide generalization bounds that account for small sample sizes. A realistic visual-servoing problem is used to illustrate the benefits of using a regularized procedure.

Approximate dynamic programming with a fuzzy parameterization

“…A rich body of literature concerns the analysis of approximate value iteration, both in the DP (model-based) setting [10,14,22,23,26] and in the RL (model-free) setting [1,12,25]. In many cases, convergence is ensured by using linearly parameterized approximators [14,25,26].…”

“…Such discretizations sometimes use interpolation schemes similar to fuzzy approximation. A different class of results analyzes the performance of approximate value iteration for stochastic processes, when only a limited number of samples are available [1,12,22].…”

“…where Q h * is the Q-function of the policy h * that is greedy in F (θ * ) (13), and Qĥ * is the Q-function of the policyĥ * that is greedy in F ( θ * ) (12).…”

“…Many regression techniques can be employed, each incurring a different computational cost. Consider e.g., the kernelbased regression of regularized fitted Q-iteration [12]. For this algorithm, the number of parameters computed in an iteration is identical to the number of samples N s employed in that iteration.…”

“…In this case, our analysis does not apply. Approaches to analyze the effect of the finite-sampling errors have been proposed in [1,12] for variants of approximate Q-iteration, and in [22] for approximate V-iteration. In some limited special cases, e.g., when there is a finite number of possible successor states, the expectation can be computed exactly, and then our results apply.…”