Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Khamaru, Koulik; Pananjady, Ashwin; Ruan, Feng; Wainwright, Martin J.; Jordan, Michael I.

doi:10.1137/20m1331524

Cited by 15 publications

(7 citation statements)

References 24 publications

(41 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this non-asymptotic setting, much of this work is focused on either the tabular case, or the simpler setting of linear function approximation, as opposed to the non-parametric cases of interest here. We note that our results do depend on the problem instance, but this instance-dependence is not (yet) as sharp as that established in the simpler setting of tabular problems [31,19].…”

Section: Related Workcontrasting

confidence: 69%

“…Our study leaves open a number of intriguing questions; let us mention a few of them here to conclude. First, although our bounds are instance-dependent, this dependence is not as refined as recent results in the simpler tabular and linear function settings [19,25]. In particular, our current results do not explicitly track the mixing properties of the transition kernel, which should enter in any such refined analysis.…”

Section: Discussionmentioning

confidence: 67%

“…It is also worth noting that recent years have witnessed considerable progress in understanding policy evaluation in off-policy settings, and/or providing guarantees that have optimal instancedependent rates. This work can be separated into work that is either asymptotic [16,17,18] and non-asymptotic [31,19,47,49] in nature. In this non-asymptotic setting, much of this work is focused on either the tabular case, or the simpler setting of linear function approximation, as opposed to the non-parametric cases of interest here.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Optimal policy evaluation using kernel-based temporal difference methods

Duan,

Wang,

Wainwright

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We study methods based on reproducing kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process (MRP). We study a regularized form of the kernel least-squares temporal difference (LSTD) estimate; in the population limit of infinite data, it corresponds to the fixed point of a projected Bellman operator defined by the associated reproducing kernel Hilbert space. The estimator itself is obtained by computing the projected fixed point induced by a regularized version of the empirical operator; due to the underlying kernel structure, this reduces to solving a linear system involving kernel matrices. We analyze the error of this estimate in the L 2 (µ)-norm, where µ denotes the stationary distribution of the underlying Markov chain. Our analysis imposes no assumptions on the transition operator of the Markov chain, but rather only conditions on the reward function and population-level kernel LSTD solutions. We use empirical process theory techniques to derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator, as well as the instance-dependent variance of the Bellman residual error. In addition, we prove minimax lower bounds over sub-classes of MRPs, which shows that our rate is optimal in terms of the sample size n and the effective horizon H = (1 − γ) −1 . Whereas existing worstcase theory predicts cubic scaling (H 3 ) in the effective horizon, our theory reveals that there is in fact a much wider range of scalings, depending on the kernel, the stationary distribution, and the variance of the Bellman residual error. Notably, it is only parametric and near-parametric problems that can ever achieve the worst-case cubic scaling.

show abstract

Section: Related Workcontrasting

confidence: 69%

Section: Discussionmentioning

confidence: 67%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Optimal policy evaluation using kernel-based temporal difference methods

Duan,

Wang,

Wainwright

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…• Policy evaluation for AMDPs (Critic): We first propose a simple and novel multiple trajectory method for policy evaluation in the generative model, which achieves O(t mix log(1/ )) sample complexity for ∞ -bound on the bias of the estimators, as well as O(t 2 mix / ) sample complexity for the expected squared ∞ -error of the estimators. For the on-policy evaluation under Markovian noise, we develop an average-reward variant of the variance-reduced temporal difference (VRTD) algorithm (Khamaru et al, 2021;Li et al, 2021) with linear function approximation, which achieves O(t 3 mix log(1/ )) sample complexity for weighted 2error of the bias of the estimators, as well as an instance-dependent sample complexity for expected weighted 2 -error of the estimators. The latter sample complexity improved the one in Zhang et al (2021b) by a factor of O(t 2 mix ).…”

Section: Main Contributionsmentioning

confidence: 99%

Stochastic first-order methods for average-reward Markov decision processes

Li¹,

Wu²,

Lan³

2022

Preprint

View full text Add to dashboard Cite

We study the problem of average-reward Markov decision processes (AMDPs) and develop novel first-order methods with strong theoretical guarantees for both policy evaluation and optimization. Existing on-policy evaluation methods suffer from sub-optimal convergence rates as well as failure in handling insufficiently random policies, e.g., deterministic policies, for lack of exploration. To remedy these issues, we develop a novel variance-reduced temporal difference (VRTD) method with linear function approximation for randomized policies along with optimal convergence guarantees, and an exploratory variance-reduced temporal difference (EVRTD) method for insufficiently random policies with comparable convergence guarantees. We further establish linear convergence rate on the bias of policy evaluation, which is essential for improving the overall sample complexity of policy optimization. On the other hand, compared with intensive research interest in finite sample analysis of policy gradient methods for discounted MDPs, existing studies on policy gradient methods for AMDPs mostly focus on regret bounds under restrictive assumptions on the underlying Markov processes (see, e.g., Abbasi-Yadkori et al., 2019), and they often lack guarantees on the overall sample complexities. Towards this end, we develop an average-reward variant of the stochastic policy mirror descent (SPMD) (Lan, 2022). We establish the first O( −2 ) sample complexity for solving AMDPs with policy gradient method under both the generative model (with unichain assumption) and Markovian noise model (with ergodic assumption). This bound can be further improved to O( −1 ) for solving regularized AMDPs. Our theoretical advantages are corroborated by numerical experiments.

show abstract

“…However, the consequence of accumulating gradients is that as the training increases, the step size decreases and eventually the training stalls. To address the training stagnation problem that occurs with AdaGrad, Hinton, G. proposed RMSProp (Root Mean Square Prop) to calculate the cumulative gradient using a moving average method that only accumulates the gradient of one window, making the change in step size adapt to the current gradient thus achieving better optimization [7,8]. And for SGD stochasticity introduces the variance problem, where SVRG is used to correct the gradient used for each model update using the global average gradient information [9].…”

Section: Introductionmentioning

confidence: 99%

N-SVRG: Stochastic Variance Reduction Gradient with Noise Reduction Ability for Small Batch Samples

Pan¹,

Zheng²

2022

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

The machine learning model converges slowly and has unstable training since large variance by random using a sample estimate gradient in SGD. To this end, we propose a noise reduction method for Stochastic Variance Reduction gradient (SVRG), called N-SVRG, which uses small batches samples instead of all samples for the average gradient calculation, while performing an incremental update of the average gradient. In each round of iteration, a small batch of samples is randomly selected for the average gradient calculation, while the average gradient is updated by rounding of the past model gradients during internal iterations. By suitably reducing the batch size B, the memory storage as well as the number of iterations can be reduced. The experiments are compared with the state-of-the-art Mini-Batch SGD, AdaGrad, RMSProp, SVRG and SCSG, and it is demonstrated that N-SVRG outperforms SVRG and SASG, and is on par with SCSG. Finally, by exploring the relationship between the small values of different parameters n. B and k and the effectiveness of the algorithm, we prove that our N-SVRG algorithm has some stability and can achieve sufficient accuracy even in the case of small batch size. The advantages and disadvantages of various methods are experimentally compared, and the stability of N-SVRG is explored by parameter settings.

show abstract

Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Cited by 15 publications

References 24 publications

Optimal policy evaluation using kernel-based temporal difference methods

Optimal policy evaluation using kernel-based temporal difference methods

Stochastic first-order methods for average-reward Markov decision processes

N-SVRG: Stochastic Variance Reduction Gradient with Noise Reduction Ability for Small Batch Samples

Contact Info

Product

Resources

About