Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning

Abstract

Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected. Towards bridging this gap, this paper initiates the theoretical study of policy finetuning, that is, online RL where the learner has additional access to a "reference policy" µ close to the optimal policy π ⋆ in a certain sense. We consider the policy finetuning problem in episodic Markov Decision Processes (MDPs) with S states, A actions, and horizon length H. We first design a sharp offline reduction algorithmwhich simply executes µ and runs offline policy optimization on the collected dataset-that finds an ε near-optimal policy within O(H 3 SC ⋆ /ε 2 ) episodes, where C ⋆ is the single-policy concentrability coefficient between µ and π ⋆ . This offline result is the first that matches the sample complexity lower bound in this setting, and resolves a recent open question in offline RL. We then establish an Ω(H 3 S min{C ⋆ , A}/ε 2 ) sample complexity lower bound for any policy finetuning algorithm, including those that can adaptively explore the environment. This implies that-perhaps surprisingly-the optimal policy finetuning algorithm is either offline reduction or a purely online RL algorithm that does not use µ. Finally, we design a new hybrid offline/online algorithm for policy finetuning that achieves better sample complexity than both vanilla offline reduction and purely online RL algorithms, in a relaxed setting where µ only satisfies concentrability partially up to a certain time step. Overall, our results offer a quantitative understanding on the benefit of a good reference policy, and make a step towards bridging offline and online RL.

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity

“…up to log factor) for small enough ε (namely, ε ≤ (0, 1/H]). The ε-range that enjoys nearoptimality is much larger compared to ε ≤ 0, 1/H 2.5 established in Xie et al (2021b) for model-based algorithms.…”

“…We prove that pessimistic Q-learning finds an ε-optimal policy as soon as the sample size T exceeds the order of (up to log factor) H 6 SC ε 2 , where C denotes the single-policy concentrability coefficient of the batch dataset. In comparison to the minimax lower bound Ω H 4 SC ε 2 developed in Xie et al (2021b), the sample complexity of pessimistic Q-learning is at most a factor of H 2 from optimal (modulo some log factor).…”

“…In this paper, we consider finite-horizon non-stationary Markov decision processes (MDPs) with S states, A actions, and horizon length H. The focal point is to pin down the sample efficiency for pessimistic variants of model-free algorithms, under the mild single-policy concentrability assumption (cf. Assumption 1) of the Xie et al (2021b) (in short, this assumption captures how close the batch dataset is to an expert dataset, and will be formally introduced in Section 2.2). Given K episodes of history data each of length H (which amounts to a total number of T = KH samples), our main contributions are summarized as follows.…”

“…Recently, the principle of pessimism (or conservatism) -namely, being conservative in Q-function estimation when there are not enough samples -has been put forward as an effective way to solve offline RL (Buckman et al, 2020;Kumar et al, 2020). This principle has been implemented in, for instance, a model-based offline value iteration algorithm, which modifies classical value iteration (Azar et al, 2017) by subtracting a penalty term in the estimated Q-values and has been shown to achieve appealing sample efficiency (Jin et al, 2021;Rashidinejad et al, 2021;Xie et al, 2021b). It is noteworthy that the model-based approach is built upon the construction of an empirical transition kernel, and therefore, requires specific representation of the environment (see, e.g.…”

Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies

“…We believe this is likely most effective when one has a lot of domain knowledge of the task, and when being applied to tasks that are too difficult for the algorithm to learn initially. For example, the authors of [25] use this approach when controlling Cassie. This approach works well for them because they are able to engineer a reward that led to the desired behavior.…”