Instance-Dependent Confidence and Early Stopping for Reinforcement Learning

Abstract

Various algorithms for reinforcement learning (RL) exhibit dramatic variation in their convergence rates as a function of problem structure. Such problem-dependent behavior is not captured by worst-case analyses and has accordingly inspired a growing effort in obtaining instance-dependent guarantees and deriving instance-optimal algorithms for RL problems. This research has been carried out, however, primarily within the confines of theory, providing guarantees that explain ex post the performance differences observed. A natural next step is to convert these theoretical guarantees into guidelines that are useful in practice. We address the problem of obtaining sharp instance-dependent confidence regions for the policy evaluation problem and the optimal value estimation problem of an MDP, given access to an instance-optimal algorithm. As a consequence, we propose a data-dependent stopping rule for instance-optimal algorithms. The proposed stopping rule adapts to the instance-specific difficulty of the problem and allows for early termination for problems with favorable structure.

A Framework for History-Aware Hyperparameter Optimisation in Reinforcement Learning

“…The feedback obtained from the proposed framework can be further exploited for example for safe early stopping [9]. Moreover, by choosing a set of high-level operations from hyperparameter tuning to algorithm selection set, to guide an agent to perform various tasks, like remembering history, comparing and contrasting current and past inputs, and using learning methods to change its own learning methods, the proposed approach can be considered a first step towards Meta-Learning [27].…”