Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?

Abstract

Learning to plan for long horizons is a central challenge in episodic reinforcement learning problems. A fundamental question is to understand how the difficulty of the problem scales as the horizon increases. Here the natural measure of sample complexity is a normalized one: we are interested in the number of episodes it takes to provably discover a policy whose value is ε near to that of the optimal value, where the value is measured by the normalized cumulative reward in each episode. In a COLT 2018 open problem, Jiang and Agarwal conjectured that, for tabular, episodic reinforcement learning problems, there exists a sample complexity lower bound which exhibits a polynomial dependence on the horizon -a conjecture which is consistent with all known sample complexity upper bounds. This work refutes this conjecture, proving that tabular, episodic reinforcement learning is possible with a sample complexity that scales only logarithmically with the planning horizon. In other words, when the values are appropriately normalized (to lie in the unit interval), this results shows that long horizon RL is no more difficult than short horizon RL, at least in a minimax sense.Our analysis introduces two ideas: (i) the construction of an ε-net for optimal policies whose log-covering number scales only logarithmically with the planning horizon, and (ii) the Online Trajectory Synthesis algorithm, which adaptively evaluates all policies in a given policy class using sample complexity that scales with the log-covering number of the given policy class. Both may be of independent interest.

Settling the Horizon-Dependence of Sample Complexity in Reinforcement Learning

“…The result in [WDYK20] shows a surprising fact that one can achieve similar sample efficiency regardless of the horizon length of the RL problem. A number of recent results [ZJD20, MDSV21, ZDJ20, RLD + 21] additionally improve the result in [WDYK20] to obtain more (computationally or statistically) efficient algorithm and/or in more general settings. However, these results all have the poly((log H)/ǫ) factor in their sample complexity, seemingly implying that the polylog(H) dependence is necessary.…”

“…Under , where the second term still scales polynomially with H. Wang et al [WDYK20] show that it is possible to obtain an ǫ-optimal policy with a sample complexity of poly(|S||A|/ǫ) • log 3 H, establishing the first sample complexity with polylog(H) dependence on the horizon length H. They achieve this result by using the following ideas: (1) samples collected by different policies can be reused to evaluate other policies; (2) to evaluate all policies in a finite set Π, the number of sample episodes required is at most poly(|S||A|/ǫ) • log |Π| • log 2 H; (3) establish a set of policies Π that contains at least one ǫ-optimal policy for any MDP by using an ǫ-nets over the reward values and the transition probabilities. Here Π contains all optimal non-stationary policies of each MDP in the ǫ-net.…”

“…Jiang and Agarwal [JA18] conjecture that the number of episodes required by the above problem depend polynomially on H even when the total number of state-action pairs is a constant. Wang et al [WDYK20] refute this conjecture by showing that the correct sample complexity of episodic RL is at most poly((log H)/ǫ) when there are constant number of states and actions. The result in [WDYK20] shows a surprising fact that one can achieve similar sample efficiency regardless of the horizon length of the RL problem.…”

“…Wang et al [WDYK20] refute this conjecture by showing that the correct sample complexity of episodic RL is at most poly((log H)/ǫ) when there are constant number of states and actions. The result in [WDYK20] shows a surprising fact that one can achieve similar sample efficiency regardless of the horizon length of the RL problem. A number of recent results [ZJD20, MDSV21, ZDJ20, RLD + 21] additionally improve the result in [WDYK20] to obtain more (computationally or statistically) efficient algorithm and/or in more general settings.…”

First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approach

“…Worst-Case Regret Bounds in Tabular RL. A significant amount of work has been devoted to obtaining worst-case optimal bounds in the setting of tabular RL (Kearns & Singh, 2002;Kakade, 2003;Azar et al, 2017;Dann et al, 2017;Jin et al, 2018;Dann et al, 2019;Wang et al, 2020;Zhang et al, 2020b,a). These approaches fall into both the model-based (Azar et al, 2017;Dann et al, 2017) as well as the model-free category (Jin et al, 2018).…”