Accelerated and instance-optimal policy evaluation with linear function approximation

Abstract

We study the problem of policy evaluation with linear function approximation and present efficient and practical algorithms that come with strong optimality guarantees. We begin by proving lower bounds that establish baselines on both the deterministic error and stochastic error in this problem. In particular, we prove an oracle complexity lower bound on the deterministic error in an instance-dependent norm associated with the stationary distribution of the transition kernel, and use the local asymptotic minimax machinery to prove an instance-dependent lower bound on the stochastic error in the i.i.d. observation model. Existing algorithms fail to match at least one of these lower bounds: To illustrate, we analyze a variance-reduced variant of temporal difference learning, showing in particular that it fails to achieve the oracle complexity lower bound. To remedy this issue, we develop an accelerated, variance-reduced fast temporal difference algorithm (VRFTD) that simultaneously matches both lower bounds and attains a strong notion of instance-optimality. Finally, we extend the VRFTD algorithm to the setting with Markovian observations, and provide instance-dependent convergence results that match those in the i.i.d. setting up to a multiplicative factor that is proportional to the mixing time of the chain. Our theoretical guarantees of optimality are corroborated by numerical experiments.1 Two of these observation models are formally discussed in Section 2.

Stochastic first-order methods for average-reward Markov decision processes

“…It is an instance-dependent characterization of the stochastic error for solving the projected Bellman equation (3.13) with sample access to the transition kernel. A direct extension of Proposition 1 of Li et al (2021) demonstrates that this term matches the asymptotic instance-dependent lower bound on stochastic error of solving Eq. (3.13) with i.i.d.…”

“…• Policy evaluation for AMDPs (Critic): We first propose a simple and novel multiple trajectory method for policy evaluation in the generative model, which achieves O(t mix log(1/ )) sample complexity for ∞ -bound on the bias of the estimators, as well as O(t 2 mix / ) sample complexity for the expected squared ∞ -error of the estimators. For the on-policy evaluation under Markovian noise, we develop an average-reward variant of the variance-reduced temporal difference (VRTD) algorithm (Khamaru et al, 2021;Li et al, 2021) with linear function approximation, which achieves O(t 3 mix log(1/ )) sample complexity for weighted 2error of the bias of the estimators, as well as an instance-dependent sample complexity for expected weighted 2 -error of the estimators. The latter sample complexity improved the one in Zhang et al (2021b) by a factor of O(t 2 mix ).…”

“…On the other hand, the data generated by the Markovian noise model is highly correlated, which induces challenges in the algorithmic analysis. To overcome the difficulties, we incorporate the idea of sample skipping from Kotsalis et al (2020b); Li et al (2021) to reduce the bias induced by correlated data. The spirit of this sample skipping idea can be traced back to the classical work by Yu (1994).…”

“…We introduce the basic idea of Algorithm 2 below. First, it incorporates the idea of the variancereduced TD learning for solving DMDPs from Li et al (2021). In particular, this algorithm runs in epochs and utilizes the current best estimator θ to recenter the updates inside each epoch.…”

Optimal variance-reduced stochastic approximation in Banach spaces

“…For stochastic gradient (SG) methods in the Euclidean setting, such bounds have been established for Polyak-Ruppert-averaged SG [MB11, GP17] and variancereduced SG algorithms [FGKS15,LMWJ20], with the sample complexity and high-order terms being improved over time. For reinforcement learning problems, such type of guarantees have been established in the • ∞ norm for temporal difference methods [KPR + 20] and Q-learning [KXWJ21] under a generative model, as well as Markovian trajectories [MPWB21,LLP21] under the ℓ 2 -norm. In the context of stochastic optimization, the paper [LMWJ20] provides fine-grained bound for ROOT-SGD with a unity pre-factor on the leading-order instancedependent term.…”