Finite-Time Analysis of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learning

Zeng, Sihan; Doan, Thinh T.; Romberg, Justin

doi:10.48550/arxiv.2010.15088

Cited by 5 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first three papers here deal with the distributed TD(0) method. These show that a result similar to the one in Zeng et al [2020b] holds for this method under constant stepsizes. In contrast, when α n is of the 1/n type, it is proven that E…”

Section: Related Worksupporting

confidence: 76%

“…Finite-time literature, in contrast, majorly talks about expectation bounds. Assuming there exists a unique x * that solves m i=1 h i (x) = 0, these results describe the rate at which E x n − x * decays with n. Notable contributions for DSA here are Zeng et al [2020b], Wai [2020]. Compared to ours, these look at a slightly different setup: the measurement noise at each node has a Markov component in place of a martingale difference term.…”

Section: Related Workmentioning

confidence: 77%

“…Compared to ours, these look at a slightly different setup: the measurement noise at each node has a Markov component in place of a martingale difference term. In this new setup, Zeng et al [2020b] shows that, for any sufficiently small but constant stepsize ǫ, the expected error decreases linearly to a ball of radius O( ǫ ln(1/ǫ)). On the other hand, Wai [2020] deals with the case where h is additionally nonconvex and shows that E x n − x * = O(n −1/4 √ ln n), which is comparable to the best known bound in the centralized setting.…”

Section: Related Workmentioning

confidence: 88%

See 2 more Smart Citations

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Thoppe¹,

Kumar²

2021

Preprint

View full text Add to dashboard Cite

In Multi-Agent Reinforcement Learning (MARL), multiple agents interact with a common environment, as also with each other, for solving a shared problem in sequential decision-making. It has wide-ranging applications in gaming, robotics, finance, etc. In this work, we derive a novel law of iterated logarithm for a family of distributed nonlinear stochastic approximation schemes that is useful in MARL. In particular, our result describes the convergence rate on almost every sample path where the algorithm converges. This result is the first of its kind in the distributed setup and provides deeper insights than the existing ones, which only discuss convergence rates in the expected or the CLT sense. Importantly, our result holds under significantly weaker assumptions: neither the gossip matrix needs to be doubly stochastic nor the stepsizes square summable. As an application, we show that, for the stepsize n −γ with γ ∈ (0, 1), the distributed TD(0) algorithm with linear function approximation has a convergence rate of O( √ n −γ ln n) a.s.; for the 1/n type stepsize, the same is O( √ n −1 ln ln n) a.s. These decay rates do not depend on the graph depicting the interactions among the different agents.Preprint. Under review.

show abstract

Section: Related Worksupporting

confidence: 76%

Section: Related Workmentioning

confidence: 77%

Section: Related Workmentioning

confidence: 88%

See 1 more Smart Citation

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Thoppe¹,

Kumar²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This can be viewed as an application of ideas from distributed stochastic approximation [34][35][36][37][38][39]. Finite-time performance guarantees for distributed RL have also been provided in works, most notably in [3,[40][41][42][43][44].…”

Section: Related Workmentioning

confidence: 99%

“…The assumption that is the central concern of this paper and is made in all the existing finite-time analyses for distributed RL algorithms is that the consensus interaction is characterized by doubly stochastic matrices [3,[40][41][42][43][44] at every time step, or at least in expectation, i.e., W 1 = 1 and 1 ⊤ E(W ) = 1 ⊤ [38]. Intuitively, doubly stochastic matrices imply symmetry in the communication graph, which almost always requires bidirectional communication graphs.…”

Section: Related Workmentioning

confidence: 99%

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Ye¹,

Gupta²,

Liu³

2021

Preprint

View full text Add to dashboard Cite

This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm.

show abstract

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Zeng¹,

Doan²,

Romberg³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We study a novel two-time-scale stochastic gradient method for solving optimization problems where the gradient samples are generated from a time-varying Markov random process parameterized by the underlying optimization variable. These time-varying samples make the stochastic gradient biased and dependent, which can potentially lead to the divergence of the iterates. To address this issue, we consider a two-time-scale update scheme, where one scale is used to estimate the true gradient from the Markovian samples and the other scale is used to update the decision variable with the estimated gradient. While these two iterates are implemented simultaneously, the former is updated "faster" (using bigger step sizes) than the latter (using smaller step sizes). Our first contribution is to characterize the finite-time complexity of the proposed two-time-scale stochastic gradient method. In particular, we provide explicit formulas for the convergence rates of this method under different objective functions, namely, strong convexity, convexity, non-convexity under the PŁ condition, and general non-convexity.Our second contribution is to apply our framework to study the performance of the popular actor-critic methods in solving stochastic control and reinforcement learning problems. First, we study an online natural actor-critic algorithm for the linear-quadratic regulator and show that a convergence rate of Opk ´2{3 q is achieved. This is the first time such a result is known in the literature. Second, we look at the standard online actor-critic algorithm over finite state and action spaces and derive a convergence rate of Opk ´2{5 q, which recovers the best known rate derived specifically for this problem. Finally, we support our theoretical analysis with numerical simulations where the convergence rate is visualized.

show abstract

Finite-Time Analysis of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learning

Cited by 5 publications

References 0 publications

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Contact Info

Product

Resources

About