Model selection in reinforcement learning

Abstract

We consider the problem of model selection in the batch (offline, non-interactive) reinforcement learning setting when the goal is to find an action-value function with the smallest Bellman error among a countable set of candidates functions. We propose a complexity regularization-based model selection algorithm, BERMIN, and prove that it enjoys an oracle-like property: the estimator's error differs from that of an oracle, who selects the candidate with the minimum Bellman error, by only a constant factor and a small remainder term that vanishes at a parametric rate as the number of samples increases. As an application, we consider a problem when the true action-value function belongs to an unknown member of a nested sequence of function spaces. We show that under some additional technical conditions BERMIN leads to a procedure whose rate of convergence, up to a constant factor, matches that of an oracle who knows which of the nested function spaces the true action-value function belongs to, i.e., the procedure achieves adaptivity.

Model-Selection for Non-parametric Function Approximation in Continuous Control Problems: A Case Study in a Smart Energy System

“…Our experiments are run using GridLAB-D 5 , an open-source smart-grid simulator that was developed for the U.S. Dept. of Energy.…”

“…There, the setup was offline, supervised learning for learning the transition function, while ours is an online reinforcement learning setup, for approximating the value function, where there are no labels over the data, but only the values to which FVI converge to, which could be different then the real state values. A paper that is closely related to ours is [5], which designs an abstract model-selection algorithm and proves theoretical guarantees about it. Similarly to here, they consider batch RL, in which a data set D of sampled transitions from the MDP is given, and is used for selecting a candidate value function by minimizing a Bellman error.…”

Feature selection and feature learning for high-dimensional batch reinforcement learning: A survey

“…Farahmand et al [36] presented a regularized fitted Q iteration algorithm based on L2 regularization to control the complexity of the value function. Farahmand and Szepesvári [37] developed a complexity regularization-based algorithm to solve the problem of model selection in the batch RL algorithms, which was formulated as finding an action-value function with a small Bellman error among a set of candidate functions. The L2 regularized LSTD problem is presented by adding an L2 penalty term into the projection equation (16) …”

Classification-Based Approximate Policy Iteration

“…If Π matches the regularity of the policy, we achieve better error upper bounds. PolicyEval and Π should ideally be chosen by an automatic model selection algorithm [25].…”