Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

Abstract

We consider the problem of fully decentralized multi-agent reinforcement learning (MARL), where the agents are located at the nodes of a time-varying communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network.Within this setting, the collective goal of the agents is to maximize the globally averaged return over the network through exchanging information with their neighbors. To this end, we propose two decentralized actor-critic algorithms with function approximation, which are applicable to large-scale MARL problems where both the number of states and the number of agents are massively large. Under the decentralized structure, the actor step is performed individually by each agent with no need to infer the policies of others. For the critic step, we propose a consensus update via communication over the network. Our algorithms are fully incremental and can be implemented in an online fashion. Convergence analyses of the algorithms are provided when the value functions are approximated within the class of linear functions. Extensive simulation results with both linear and nonlinear function approximations are presented to validate the proposed algorithms. Our work appears to be the first study of fully decentralized MARL algorithms for networked agents with function approximation, with provable convergence guarantees.

BotNet: A Simulator for Studying the Effects of Accurate Communication Models on Multi-agent and Swarm Control

“…The potential of peer-to-peer communication is highlighted by progress in decentralized control, but recent work has incorporated only a limited study of realistic propagation models, schedule functions, multi-hop behavior, and other real-world aspects of complex networks. The simplest network abstractions used in robotics studies focus on swarms as a graph structure, using a model where the nearest neighbors can communicate fully [25], [26]. Some work uses line-of-sight (LOS) instead of purely geometric distance in order to determine which agents can communicate [27], [28], but LOS is not an accurate model for RF networks.…”

Approximated Multi-Agent Fitted Q Iteration

“…i.e., qj N (x, a) is an approximation of the centralized Qfunction's maximum when agent j uses the control u(j) = a. We re-express (8) and let Qj N be a monotonically increasing approximation of the Q-function for agent j after N iterations given the available batch data that satisfies:…”

“…In multi-agent reinforcement learning, agents make sequential decisions to maximize their joint or individual rewards [7], [8]. The agents can be fully cooperative, i.e., maximizing a joint reward function, fully competitive, i.e., the agents' objectives are opposed, or a combination of both [7], [8].…”

“…In multi-agent reinforcement learning, agents make sequential decisions to maximize their joint or individual rewards [7], [8]. The agents can be fully cooperative, i.e., maximizing a joint reward function, fully competitive, i.e., the agents' objectives are opposed, or a combination of both [7], [8]. The main challenge when considering the multi-agent reinforcement learning problem comes from the cardinality of the joint control set as it increases exponentially with the number of agents.…”

“…In our work, we are interested in multiagent extensions of batch reinforcement learning [2], and more specifically, of the kernel-based [3] FQI [4] framework. Multiagent problems have also been studied under other reinforcement learning frameworks, e.g., classical Q-learning [22], [23] or actor-critic approaches [8], [24]. We review the literature relevant to multi-agent FQI next.…”