Instance-optimality in optimal value estimation: Adaptivity via variance-reduced Q-learning

Khamaru, Koulik; Xia, Eric; Wainwright, Martin J.; Jordan, Michael I.

doi:10.48550/arxiv.2106.14352

Cited by 8 publications

(42 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…, this matches the two-point lower bound in the paper [KXWJ21] (in the discounted MDP case). When specializing to the cases where the optimal policy is unique, or satisfies the Lipschitz-type assumptions in the paper [KXWJ21], the upper bound above also recovers the leading-order term in that paper. We conjecture that the leading-order term of the solution s n to the fixed-point equation is actually optimal for large n. It is an important direction of future work to investigate this gap, and establish optimality results under suitably defined problem classes.…”

Section: Guarantees For Stochastic Shortest Pathmentioning

confidence: 55%

“…Note that when specialized to the γ-discounted MDPs, the sample size requirement in Corollary 5 becomes O (1 − γ) −4 , which can be worse than the corresponding requirements in the paper [KXWJ21], at least in certain regimes. Intuitively, this is the price we pay when moving to the general case where only the contraction of the population-level operator is assumed, instead of the sample-level contraction.…”

Section: Guarantees For Stochastic Shortest Pathmentioning

confidence: 89%

“…For stochastic gradient (SG) methods in the Euclidean setting, such bounds have been established for Polyak-Ruppert-averaged SG [MB11, GP17] and variancereduced SG algorithms [FGKS15,LMWJ20], with the sample complexity and high-order terms being improved over time. For reinforcement learning problems, such type of guarantees have been established in the • ∞ norm for temporal difference methods [KPR + 20] and Q-learning [KXWJ21] under a generative model, as well as Markovian trajectories [MPWB21,LLP21] under the ℓ 2 -norm. In the context of stochastic optimization, the paper [LMWJ20] provides fine-grained bound for ROOT-SGD with a unity pre-factor on the leading-order instancedependent term.…”

Section: Stochastic Approximation and Asymptotic Guaranteesmentioning

confidence: 99%

“…The sample size requirement may depend on the gap between the value of optimal and sub-optimal actions, as in the prior work[KXWJ21].…”

mentioning

confidence: 99%

“…The sample size requirement for achieving the lower bound[KXWJ21] may depend on the gap between the value of optimal and sub-optimal actions.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Optimal variance-reduced stochastic approximation in Banach spaces

Mou¹,

Khamaru²,

Wainwright³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. Focusing on a stochastic query model that provides noisy evaluations of the operator, we analyze a variance-reduced stochastic approximation scheme, and establish non-asymptotic bounds for both the operator defect and the estimation error, measured in an arbitrary semi-norm. In contrast to worst-case guarantees, our bounds are instance-dependent, and achieve the local asymptotic minimax risk non-asymptotically. For linear operators, contractivity can be relaxed to multi-step contractivity, so that the theory can be applied to problems like average reward policy evaluation problem in reinforcement learning. We illustrate the theory via applications to stochastic shortest path problems, two-player zero-sum Markov games, as well as policy evaluation and Q-learning for tabular Markov decision processes.

show abstract

Section: Guarantees For Stochastic Shortest Pathmentioning

confidence: 55%

Section: Guarantees For Stochastic Shortest Pathmentioning

confidence: 89%

Section: Stochastic Approximation and Asymptotic Guaranteesmentioning

confidence: 99%

“…The sample size requirement may depend on the gap between the value of optimal and sub-optimal actions, as in the prior work[KXWJ21].…”

mentioning

confidence: 99%

“…The sample size requirement for achieving the lower bound[KXWJ21] may depend on the gap between the value of optimal and sub-optimal actions.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Optimal variance-reduced stochastic approximation in Banach spaces

Mou¹,

Khamaru²,

Wainwright³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Khamaru¹,

Pananjady²,

Ruan³

et al. 2021

SIAM Journal on Mathematics of Data Science

View full text Add to dashboard Cite

We consider the problem of stochastic convex optimization under convex constraints. We analyze the behavior of a natural variance reduced proximal gradient (VRPG) algorithm for this problem. Our main result is a non-asymptotic guarantee for VRPG algorithm. Contrary to minimax worst case guarantees, our result is instance-dependent in nature. This means that our guarantee captures the complexity of the loss function, the variability of the noise, and the geometry of the constraint set. We show that the non-asymptotic performance of the VRPG algorithm is governed by the scaled distance (scaled by √ N ) between the solutions of the given problem and that of a certain small perturbation of the given problem -both solved under the given convex constraints; here, N denotes the number of samples. Leveraging a well-established connection between local minimax lower bounds and solutions to perturbed problems, we show that as N → ∞, the VRPG algorithm achieves the renowned local minimax lower bound by Hàjek and Le Cam up to universal constants and a logarithmic factor of the sample size.

show abstract

Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

Wagenmaker¹,

Simchowitz²,

Jamieson³

2021

Preprint

View full text Add to dashboard Cite

The theory of reinforcement learning has focused on two fundamental problems: achieving low regret, and identifying -optimal policies. While a simple reduction allows one to apply a low-regret algorithm to obtain an -optimal policy and achieve the worst-case optimal rate, it is unknown whether low-regret algorithms can obtain the instance-optimal rate for policy identification. We show that this is not possible-there exists a fundamental tradeoff between achieving low regret and identifying an -optimal policy at the instance-optimal rate. Motivated by our negative finding, we propose a new measure of instance-dependent sample complexity for PAC tabular reinforcement learning which explicitly accounts for the attainable state visitation distributions in the underlying MDP. We then propose and analyze a novel, planning-based algorithm which attains this sample complexity-yielding a complexity which scales with the suboptimality gaps and the "reachability" of a state. We show that our algorithm is nearly minimax optimal, and on several examples that our instance-dependent sample complexity offers significant improvements over worst-case bounds.

show abstract

Instance-optimality in optimal value estimation: Adaptivity via variance-reduced Q-learning

Cited by 8 publications

References 5 publications

Optimal variance-reduced stochastic approximation in Banach spaces

Optimal variance-reduced stochastic approximation in Banach spaces

Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

Contact Info

Product

Resources

About