UCB Momentum Q-learning: Correcting the bias without forgetting

Abstract

We propose UCBMQ, Upper Confidence Bound Momentum Q-learning, a new algorithm for reinforcement learning in tabular and possibly stagedependent, episodic Markov decision process. UCBMQ is based on Q-learning where we add a momentum term and rely on the principle of optimism in face of uncertainty to deal with exploration. Our new technical ingredient of UCBMQ is the use of momentum to correct the bias that Q-learning suffers while, at the same time, limiting the impact it has on the the second-order term of the regret. For UCBMQ, we are able to guarantee a regret of at most O( √ H 3 SAT + H 4 SA) where H is the length of an episode, S the number of states, A the number of actions, T the number of episodes and ignoring terms in poly log(SAHT ). Notably, UCBMQ is the first algorithm that simultaneously matches the lower bound of Ω( √ H 3 SAT ) for large enough T and has a second-order term (with respect to the horizon T ) that scales only linearly with the number of states S.1 It is the same reason why there is an extra factor S in the first order term of the bound of UCRL algorithm by Jaksch et al. (2010). This factor is "pushed" to the second-order term by the improved analysis of Azar et al. (2017).

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

“…The runtime of Q-EarlySettled-Advantage is no larger than O(T ), which is proportional to the time taken to read the samples. This matches the computational cost of the model-free algorithm UCB-Q proposed in Jin et al (2018a), and is considerably lower than that of the UCB-M-Q algorithm in Menard et al (2021) (which has a computational cost of at least O(ST )).…”

“…This is basically un-improvable for the tabular case, since even storing the optimal Q-values alone takes O(SAH) units of space. In comparison, while Menard et al (2021) also accommodates the sample size range (17), the algorithm proposed therein incurs a space complexity of O(S 2 AH) that is S times higher than ours.…”

“…• A memory-inefficient "model-free" variant. The recent work Menard et al (2021) put forward a novel sample-efficient variant of Q-learning called UCB-M-Q, which relies on a carefully chosen momentum term for bias reduction. This algorithm is guaranteed to yield near-optimal regret O √ H 2 SAT as soon as the sample size exceeds T ≥ SApoly(H), which is a remarkable improvement vis-à-vis previous regret-optimal methods (Azar et al, 2017;Zhang et al, 2020c).…”

“…As can be seen from Table 1, the space complexity of the proposed algorithm is O(SAH), which is far more memory-efficient than both the model-based approach in Azar et al (2017) and the UCB-M-Q algorithm in Menard et al (2021) (both of these prior algorithms require S 2 AH units of space). In addition, the sample size requirement T ≥ SA poly(H) of our algorithm improves -by a factor of at least S 5 A 3upon that of any prior algorithm that is simultaneously regret-optimal and memory-efficient.…”

Nearly Optimal Policy Optimization with Stable at Any Time Guarantee

“…Our work is also closely related to another line of work on value-based methods. In particular, Azar et al (2017); Zanette and Brunskill (2019); Zhang et al (2020a,b); Menard et al (2021) have shown that the value-based methods can achieve O( √ SAH 3 K) regret upper bound, which matches the information theoretic limit. Different from these works, we are the first to prove the (nearly) optimal regret bound for policy-based methods.…”