A Finite Time Analysis of Temporal Difference Learning with Linear Function Approximation

Bhandari, Jalaj; Russo, Daniel; Singal, Raghav

doi:10.1287/opre.2020.2024

Cited by 114 publications

(303 citation statements)

References 27 publications

Supporting

Mentioning

293

Contrasting

Order By: Relevance

“…provided that an appropriate constant learning rate is adopted. We note that prior finite-sample analysis on asynchronous TD learning typically focused on (weighted) 2 estimation errors with linear function approximation [21], [22], and it is hence difficult to make fair comparisons. The recent paper [23] developed ∞ guarantees for TD learning, focusing on the synchronous settings with i.i.d.…”

Section: A Special Case: Td Learningmentioning

confidence: 99%

“…The Q-learning algorithm, originally proposed in [29], has been analyzed in the asymptotic regime by [6], [7], [14], [30] since more than two decades ago. Additionally, finite-time performance of Q-learning and its variants have been analyzed by [2], [8]- [10], [19], [31]- [34] in the tabular setting, by [21], [35]- [43] in the context of function approximations, and by [44] with nonparametric regression. In addition, [11], [12], [24], [45]- [47] studied modified Q-learning algorithms that might potentially improve sample complexities and accelerate convergence.…”

Section: A the Q-learning Algorithm And Its Variantsmentioning

confidence: 99%

“…Convergence properties of several model-free RL algorithms have been studied recently in the presence of Markovian data, including but not limited to TD learning and its variants [21], [22], [28], [52]- [60], Q-learning [35], [36], and SARSA [61]. However, these recent papers typically focused on the (weighted) 2 error rather than the ∞ risk, where the latter is often more relevant in the context of RL.…”

Section: Finite-sample Guarantees For Model-free Algorithmsmentioning

confidence: 99%

See 2 more Smart Citations

Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction

Wei

Chi

et al. 2022

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Asynchronous Q-learning aims to learn the optimal action-value function (or Q-function) of a Markov decision process (MDP), based on a single trajectory of Markovian samples induced by a behavior policy. Focusing on a γ-discounted MDP with state space S and action space A, we demonstrate that the ∞ -based sample complexity of classical asynchronous Q-learning -namely, the number of samples needed to yield an entrywise ε-accurate estimate of the Q-function -is at most on the order of 1 μ min (1−γ ) 5 ε 2 + t mix μ min (1−γ ) up to some logarithmic factor, provided that a proper constant learning rate is adopted. Here, tmix and μmin denote respectively the mixing time and the minimum state-action occupancy probability of the sample trajectory. The first term of this bound matches the sample complexity in the synchronous case with independent samples drawn from the stationary distribution of the trajectory. The second term reflects the cost taken for the empirical distribution of the Markovian trajectory to reach a steady state, which is incurred at the very beginning and becomes amortized as the algorithm runs. Encouragingly, the above bound improves upon the stateof-the-art result by a factor of at least |S||A| for all scenarios, and by a factor of at least tmix|S||A| for any sufficiently small accuracy level ε. Further, we demonstrate that the scaling on the effective horizon 1 1−γ can be improved by means of variance reduction.

show abstract

Section: A Special Case: Td Learningmentioning

confidence: 99%

Section: A the Q-learning Algorithm And Its Variantsmentioning

confidence: 99%

Section: Finite-sample Guarantees For Model-free Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction

Wei

Chi

et al. 2022

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…As has been discussed in, e.g., Reference [ 43 ], the non-asymptotic rate of convergence (the rate at which the "mean-path" of the TD or Q-learning algorithm converges) is exponential. The asymptotic rate of convergence can be analyzed by deriving asymptotic stochastic differential equation which shows a direct dependence of the asymptotic covariance of the value function estimate on the network connectivity [ 12 ].…”

Section: Consensus-based Distributed Joint Spectrum Sensing and Selectionmentioning

confidence: 99%

Distributed Spectrum Management in Cognitive Radio Networks by Consensus-Based Reinforcement Learning

Dašić

Ilić

Vučetić³

et al. 2021

Sensors

View full text Add to dashboard Cite

In this paper, we propose a new algorithm for distributed spectrum sensing and channel selection in cognitive radio networks based on consensus. The algorithm operates within a multi-agent reinforcement learning scheme. The proposed consensus strategy, implemented over a directed, typically sparse, time-varying low-bandwidth communication network, enforces collaboration between the agents in a completely decentralized and distributed way. The motivation for the proposed approach comes directly from typical cognitive radio networks’ practical scenarios, where such a decentralized setting and distributed operation is of essential importance. Specifically, the proposed setting provides all the agents, in unknown environmental and application conditions, with viable network-wide information. Hence, a set of participating agents becomes capable of successful calculation of the optimal joint spectrum sensing and channel selection strategy even if the individual agents are not. The proposed algorithm is, by its nature, scalable and robust to node and link failures. The paper presents a detailed discussion and analysis of the algorithm’s characteristics, including the effects of denoising, the possibility of organizing coordinated actions, and the convergence rate improvement induced by the consensus scheme. The results of extensive simulations demonstrate the high effectiveness of the proposed algorithm, and that its behavior is close to the centralized scheme even in the case of sparse neighbor-based inter-node communication.

show abstract

“…Error bounds for constant stepsize synchronous and asynchronous Q-learning algorithms were studied in [6] by combining the union bound and triangle inequality. Finite-time bounds for temporal difference learning for evaluating stationary policies with constant stepsize have been obtained in [47,9] under a variety of assumptions.…”

mentioning

confidence: 99%

Convergence of Recursive Stochastic Algorithms Using Wasserstein Divergence

Gupta¹,

Haskell²

2021

SIAM Journal on Mathematics of Data Science

View full text Add to dashboard Cite

This paper develops a unified framework, based on iterated random operator theory, to analyze the convergence of constant stepsize recursive stochastic algorithms (RSAs). RSAs use randomization to efficiently compute expectations, and so their iterates form a stochastic process. The key idea of our analysis is to lift the RSA into an appropriate higher-dimensional space and then express it as an equivalent Markov chain. Instead of determining the convergence of this Markov chain (which may not converge under constant stepsize), we study the convergence of the distribution of this Markov chain. To study this, we define a new notion of Wasserstein divergence. We show that if the distribution of the iterates in the Markov chain satisfy a contraction property with respect to the Wasserstein divergence, then the Markov chain admits an invariant distribution. We show that convergence of a large family of constant stepsize RSAs can be understood using this framework, and we provide several detailed examples.

show abstract

A Finite Time Analysis of Temporal Difference Learning with Linear Function Approximation

Cited by 114 publications

References 27 publications

Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction

Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction

Distributed Spectrum Management in Cognitive Radio Networks by Consensus-Based Reinforcement Learning

Convergence of Recursive Stochastic Algorithms Using Wasserstein Divergence

Contact Info

Product

Resources

About