Multi-Agent Reinforcement Learning in Stochastic Networked Systems

Abstract

We study distributed reinforcement learning (RL) for a network of agents. The objective is to find localized policies that maximize the (discounted) global reward. In general, scalability is a challenge in this setting because the size of the global state/action space can be exponential in the number of agents. Scalable algorithms are only known in cases where dependencies are local, e.g., between neighbors. In this work, we propose a Scalable Actor Critic framework that applies in settings where the dependencies are non-local and provide a finite-time error bound that shows how the convergence rate depends on the depth of the dependencies in the network. Additionally, as a byproduct of our analysis, we obtain novel finite-time convergence results for a general stochastic approximation scheme and for temporal difference learning with state aggregation that apply beyond the setting of RL in networked systems.Preprint. Under review.

Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning

“…Remark 10 The works in [Qu and Li, 2019, Qu et al, 2020a, Lin et al, 2020, Qu et al, 2020b are closely related to our contribution, as they also use decay of correlation assumptions to provably avoid the curse of dimensionality in MARL. Our contribution differs from these works in the following main ways:…”

“…To take advantage of the local structure of the network, Lin et al [2020] define a property regarding the dependence of Q π k (s, a) on the neighbors of k.…”

“…Definition 4 (Lin et al [2020]) The (c, ψ)-exponential decay property for the Q-function holds if, for any agent k ∈ K and for any (s, a), ( s, a) ∈ S × A such that s…”

“…These ideas have been applied to the MARL setting [Guestrin et al, 2001a, 2002, Sunehag et al, 2017, Rashid et al, 2018, Zhang et al, 2018a,b, Zhang and Zavlanos, 2019 and have proven successful in experiments, but lack theoretical guarantees or non-asymptotic analysis. A recent line of work has formally considered spatial decay of correlation assumptions for nearest-neighbors dynamics and designed decentralized algorithms based on policy gradient and actor-critic methods [Qu and Li, 2019, Qu et al, 2020a, Lin et al, 2020, Qu et al, 2020b, establishing non-asymptotic convergence guarantees towards a stationary point, but not towards an optimal policy. 1 An application of the same principles to the setting of mean-field MARL [Yang et al, 2018] can be found in Haotian Gu [2021], where the authors show that a neural network based version of the actor-critic algorithm can achieve global convergence.…”

Convergence Rates of Average-Reward Multi-agent Reinforcement Learning via Randomized Linear Programming

“…We further focus on the case where the communications network is a structural component of the problem setting, as in (Lowe et al, 2017;Zhang et al, 2018). However, a separate but related body of works estimate the communications architecture when agents' behavior is fixed using graph neural networks (Ahilan & Dayan, 2020;Bachrach et al, 2020;Eccles et al, 2019) or statistical tests for correlation between agents' local utilities (Lin et al, 2020;Qu et al, 2020a).…”