Sample-Efficient Reinforcement Learning with loglog(T) Switching Cost

Abstract

We study the problem of reinforcement learning (RL) with low (policy) switching costa problem well-motivated by real-life RL applications in which deployments of new policies are costly and the number of policy updates must be low. In this paper, we propose a new algorithm based on stage-wise exploration and adaptive policy elimination that achieves a regret of O( √ H 4 S 2 AT ) while requiring a switching cost of O(HSA log log T ). This is an exponential improvement over the best-known switching cost O(H 2 SA log T ) among existing methods with O(poly(H, S, A) √ T ) regret. In the above, S, A denotes the number of states and actions in an H-horizon episodic Markov Decision Process model with unknown transitions, and T is the number of steps. We also prove an information-theoretical lower bound which says that a switching cost of Ω(HSA) is required for any no-regret algorithm. As a byproduct, our new algorithmic techniques allow us to derive a reward-free exploration algorithm with an optimal switching cost of O(HSA).

Near-Optimal Adversarial Reinforcement Learning with Switching Costs

“…Specifically, for tabular MDP, [5] and [39] proposed RL algorithms that attain an Õ √ H α SAT • ln T SA δ regret with probability 1 − δ, by incurring O (H α SA ln T ) switching costs, where α = 3 and 2, respectively. Recently, [11] obtained a similar Õ( √ T ) regret with probability 1 − δ, by incurring O (HSA ln ln T ) switching costs. Moreover, for linear MDP (with ddimensional feature space), [9] and [22]…”

“…Theorem 1 shows that in adversarial RL with switching costs, the dependency on T of the best achievable regret is at least Ω(T 2/3 ). Thus, the best achieved regret (whose dependency on T is Õ( √ T )) in static RL with switching costs (in [5,11], etc) as well as adversarial RL without switching costs (in [1,3], etc) is no longer achievable. This demonstrates the fundamental challenge of switching costs in adversarial RL, and it is expected that new challenges will arise when developing provably efficient algorithms.…”

“…In such a problem, an online learner interacts with the environment sequentially over Markov decision processes (MDPs), and aims to find a desirable policy for achieving an accumulated loss (or reward). Various algorithms have been developed for RL problems and have been shown theoretically to achieve polynomial sample efficiency in [1,2,3,4,5,6,7,8,9,10,11], etc.…”

“…Among these studies, a relevant line of research is along bandit learning [18,19,20,21]. More recently, switching costs have received considerable attention in more general RL settings [5,9,22,11]. However, these studies have mainly focused on static RL, where the loss distribution is assumed to be fixed during the learning process.…”

“…This requires a careful reduction of switching under non-stationary or adversarial loss distributions. It turns out that previous approaches to reduce switching in static RL (e.g., those in [5] and [11]) are not applicable here. Specifically, the high-level idea in static RL is to switch faster at the beginning, while switch slower and slower for later episodes.…”