Active Offline Policy Selection

Abstract

This paper addresses the problem of policy selection in domains with abundant logged data, but with a very restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and healthcare domain among others. Several offpolicy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation in the real environment. To reduce this gap, we introduce a novel active offline policy selection problem formulation, which combined logged data and limited online interactions to identify the best policy. We rely on the advances in OPE to warm start the evaluation. We build upon Bayesian optimization to iteratively decide which policies to evaluate in order to utilize the limited environment interactions wisely. Many candidate policies could be proposed, thus, we focus on making our approach scalable and introduce a kernel function to model similarity between policies. We use several benchmark environments to show that the proposed approach improves upon state-of-the-art OPE estimates and fully online policy evaluation with limited budget. Additionally, we show that each component of the proposed method is important, it works well with various number and quality of OPE estimates and even with a large number of candidate policies. * equal contribution Preprint. Under review.

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning

“…This form of evaluation is closer to a worst-case analysis than the typical average performance reporting. In this case, the conservative estimation is important, since there exists no equivalent to early stopping from supervised learning in the offline RL setting: Offline policy evaluation and selection are still open problems (Hans et al, 2011;Paine et al, 2020;Konyushkova et al, 2021;Zhang et al, 2021;Fu et al, 2021), so we need to stop the policy training at some random point and deploy that policy on the real system. The above procedure is meant to simulate this random stopping, and should quantify how good the algorithm performs at least.…”