Optimal and instance-dependent guarantees for Markovian linear stochastic approximation

Mou, Wenlong; Pananjady, Ashwin; Wainwright, Martin J.

doi:10.48550/arxiv.2112.12770

Cited by 3 publications

(4 citation statements)

References 22 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For stochastic gradient (SG) methods in the Euclidean setting, such bounds have been established for Polyak-Ruppert-averaged SG [MB11, GP17] and variancereduced SG algorithms [FGKS15,LMWJ20], with the sample complexity and high-order terms being improved over time. For reinforcement learning problems, such type of guarantees have been established in the • ∞ norm for temporal difference methods [KPR + 20] and Q-learning [KXWJ21] under a generative model, as well as Markovian trajectories [MPWB21,LLP21] under the ℓ 2 -norm. In the context of stochastic optimization, the paper [LMWJ20] provides fine-grained bound for ROOT-SGD with a unity pre-factor on the leading-order instancedependent term.…”

Section: Stochastic Approximation and Asymptotic Guaranteesmentioning

confidence: 99%

Optimal variance-reduced stochastic approximation in Banach spaces

Mou¹,

Khamaru²,

Wainwright³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. Focusing on a stochastic query model that provides noisy evaluations of the operator, we analyze a variance-reduced stochastic approximation scheme, and establish non-asymptotic bounds for both the operator defect and the estimation error, measured in an arbitrary semi-norm. In contrast to worst-case guarantees, our bounds are instance-dependent, and achieve the local asymptotic minimax risk non-asymptotically. For linear operators, contractivity can be relaxed to multi-step contractivity, so that the theory can be applied to problems like average reward policy evaluation problem in reinforcement learning. We illustrate the theory via applications to stochastic shortest path problems, two-player zero-sum Markov games, as well as policy evaluation and Q-learning for tabular Markov decision processes.

show abstract

Section: Stochastic Approximation and Asymptotic Guaranteesmentioning

confidence: 99%

Optimal variance-reduced stochastic approximation in Banach spaces

Mou¹,

Khamaru²,

Wainwright³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Constant stepsize has gained popularity recently, particularly among practitioners, due to its fast initial convergence and easy hyperparameter tuning. A growing line of works studies the convergence properties of SA under constant stepsize, establishing upper bounds on the meansquared error (MSE) (Lakshminarayanan and Szepesvári 2018;Srikant and Ying 2019;Mou et al 2020Mou et al , 2021 as well as weak convergence results (Dieuleveut, Durmus, and Bach 2020;Yu et al 2021;Huo, Chen, and Xie 2023a).…”

Section: Introductionmentioning

confidence: 99%

Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inference

Huo,

Chen,

Xie

2024

AAAI

View full text Add to dashboard Cite

In this paper, we study the effectiveness of using a constant stepsize in statistical inference via linear stochastic approximation (LSA) algorithms with Markovian data. After establishing a Central Limit Theorem (CLT), we outline an inference procedure that uses averaged LSA iterates to construct confidence intervals (CIs). Our procedure leverages the fast mixing property of constant-stepsize LSA for better covariance estimation and employs Richardson-Romberg (RR) extrapolation to reduce the bias induced by constant stepsize and Markovian data. We develop theoretical results for guiding stepsize selection in RR extrapolation, and identify several important settings where the bias provably vanishes even without extrapolation. We conduct extensive numerical experiments and compare against classical inference approaches. Our results show that using a constant stepsize enjoys easy hyperparameter tuning, fast convergence, and consistently better CI coverage, especially when data is limited.

show abstract

“…Initial analysis of least squares temporal difference learning (LSTD) date back to the work of Baird [1995], Bradtke and Barto [1996], Boyan [1999] and Nedić and Bertsekas [2003]. Since then, the finite sample performance of the algorithm has been analyzed by Lazaric et al [2012], Bhandari et al [2018], Duan et al [2021] and its behavior in the offline setting studied by Yu [2010], Li et al [2021], Mou et al [2020Mou et al [ , 2021. Tu and Recht [2018] analyze on-policy LSTD for the LQR setting.…”

Section: Fqi Introduced Bymentioning

confidence: 99%

“…Extensions to Markovian data are fairly well-understood, see e.g.,Mou et al [2021],Nagaraj et al [2020].3 Note that realizability of Q π does not imply that the rewards are linearly realizable.…”

mentioning

confidence: 99%

A Sharp Characterization of Linear Estimators for Offline Policy Evaluation

Perdomo¹,

Krishnamurthy²,

Kakade³

2022

Preprint

View full text Add to dashboard Cite

Offline policy evaluation is a fundamental statistical problem in reinforcement learning that involves estimating the value function of some decision-making policy given data collected by a potentially different policy. In order to tackle problems with complex, high-dimensional observations, there has been significant interest from theoreticians and practitioners alike in understanding the possibility of function approximation in reinforcement learning. Despite significant study, a sharp characterization of when we might expect offline policy evaluation to be tractable, even in the simplest setting of linear function approximation, has so far remained elusive, with a surprising number of strong negative results recently appearing in the literature.In this work, we identify simple control-theoretic and linear-algebraic conditions that are necessary and sufficient for classical methods, in particular Fitted Q-iteration (FQI) and least squares temporal difference learning (LSTD), to succeed at offline policy evaluation. Using this characterization, we establish a precise hierarchy of regimes under which these estimators succeed. We prove that LSTD works under strictly weaker conditions than FQI. Furthermore, we establish that if a problem is not solvable via LSTD, then it cannot be solved by a broad class of linear estimators, even in the limit of infinite data. Taken together, our results provide a complete picture of the behavior of linear estimators for offline policy evaluation (OPE), unify previously disparate analyses of canonical algorithms, and provide significantly sharper notions of the underlying statistical complexity of OPE.

show abstract

Optimal and instance-dependent guarantees for Markovian linear stochastic approximation

Cited by 3 publications

References 22 publications

Optimal variance-reduced stochastic approximation in Banach spaces

Optimal variance-reduced stochastic approximation in Banach spaces

Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inference

A Sharp Characterization of Linear Estimators for Offline Policy Evaluation

Contact Info

Product

Resources

About