Distributed Zero-Order Optimization under Adversarial Noise

Akhavan, Arya; Pontil, Massimiliano; Tsybakov, Alexandre B.

doi:10.48550/arxiv.2102.01121

Cited by 2 publications

(3 citation statements)

References 24 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, under different zeroth-order oracles, we show that our learning framework always exhibits a reduced variance of the estimated gradient compared with GVF-based policy evaluation. Note that most of the existing distributed ZOO algorithms [30][31][32][33] essentially evaluate policies via GVFs. In [34,35],…”

Section: Introductionmentioning

confidence: 99%

Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value-Functions

Jing¹,

He²,

George³

et al. 2022

Preprint

View full text Add to dashboard Cite

The main challenge of large-scale cooperative multi-agent reinforcement learning (MARL) is twofold: (i) the RL algorithm is desired to be distributed due to limited resource for each individual agent; (ii) issues on convergence or computational complexity emerge due to the curse of dimensionality. Unfortunately, most of existing distributed RL references only focus on ensuring that the individual policy-seeking process of each agent is based on local information, but fail to solve the scalability issue induced by high dimensions of the state and action spaces when facing large-scale networks. In this paper, we propose a general distributed framework for cooperative MARL by utilizing the structures of graphs involved in this problem. We introduce three graphs in MARL, namely, the coordination graph, the observation graph and the reward graph. Based on these three graphs, and a given communication graph, we propose two distributed RL approaches. The first approach utilizes the inherent decomposability property of the problem itself, whose efficiency depends on the structures of the aforementioned four graphs, and is able to produce a high performance under specific graphical conditions. The second approach provides an approximate solution and is applicable for any graphs. Here the approximation error depends on an artificially designed index. The choice of this index is a trade-off between minimizing the approximation error and reducing the computational complexity. Simulations show that our RL algorithms have a significantly improved scalability to large-scale MASs compared with centralized and consensus-based distributed RL algorithms.

show abstract

Section: Introductionmentioning

confidence: 99%

Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value-Functions

Jing¹,

He²,

George³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Multi-agent networks are one of the most representative systems that have broad applications and usually induce largesize optimization problems [12]. In recent years, distributed zeroth-order convex and non-convex optimizations on multi-agent networks have been extensively studied, e.g., [13]- [17], all of which decompose the original cost function into multiple functions and assign them to the agents. Unfortunately, the variable dimension for each agent is the same as that for the original problem.…”

Section: Introductionmentioning

confidence: 99%

“…• In comparison to centralized ZOO [2]- [4], [9], [11] and distributed ZOO [13]- [17], we reduce the variable dimension for each agent, and construct local cost involving only partial agents. Such a framework avoids the influence of the convergence error and convergence rate of the consensus algorithm, and results in reduced variance and high scalability to large-scale networks.…”

Section: Introductionmentioning

confidence: 99%

Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent

Jing¹,

Bai²,

George³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the network structure inherent in the optimization objective, which allows each agent to estimate its local gradient by local cost evaluation independently, without use of any consensus protocol. The proposed algorithm exhibits an asynchronous update scheme, and is designed for stochastic non-convex optimization with a possibly non-convex feasible domain based on the block coordinate descent method. The algorithm is later employed as a distributed model-free RL algorithm for distributed linear quadratic regulator design, where a learning graph is designed to describe the required interaction relationship among agents in distributed learning. We provide an empirical validation of the proposed algorithm to benchmark its performance on convergence rate and variance against a centralized ZOO algorithm.

show abstract

Distributed Zero-Order Optimization under Adversarial Noise

Cited by 2 publications

References 24 publications

Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value-Functions

Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value-Functions

Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent

Contact Info

Product

Resources

About