A Statistical Analysis of Polyak-Ruppert Averaged Q-learning

Abstract

We study synchronous Q-learning with Polyak-Ruppert averaging (a.k.a., averaged Q-leaning) in a γ-discounted MDP. We establish asymptotic normality for the averaged iteration QT . Furthermore, we show that QT is actually a regular asymptotically linear (RAL) estimator for the optimal Q-value function Q * with the most efficient influence function. It implies the averaged Q-learning iteration has the smallest asymptotic variance among all RAL estimators. In addition, we present a non-asymptotic analysis for the ∞ error E QT − Q * ∞ , showing it matches the instance-dependent lower bound as well as the optimal minimax complexity lower bound. As a byproduct, we find the Bellman noise has sub-Gaussian coordinates with variance O((1 − γ) −1 ) instead of the prevailing O((1 − γ) −2 ) under the standard bounded reward assumption. The sub-Gaussian result has potential to improve the sample complexity of many RL algorithms. In short, our theoretical analysis shows averaged Q-Leaning is statistically efficient.

KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal

“…There exist minimax-optimal model-based algorithms for learning a near-optimal value function (Azar et al, 2013) and policy (Agarwal et al, 2020;Li et al, 2020). Also, there exist minimax-optimal model-free algorithms for learning a near-optimal value function (Wainwright, 2019;Khamaru et al, 2021;Li et al, 2021b) and policy (Sidford et al, 2018). While model-based algorithms are conceptually simple, they have a higher computational complexity than that of model-free algorithms.…”

“…MDVI's underlying idea that enables such simplicity is, while implicit, the averaging of value function estimates. Li et al (2021b) shows that averaging Q-functions computed in Q-learning can find a near-optimal Q-function with a minimax-optimal sample complexity. Azar et al (2011) also provides a simple algorithm called Speedy Q-learning (SQL), which performs the averaging of value function estimates.…”