Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Abstract

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered great success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Though empirically successful, theoretical foundations for MARL are relatively lacking in the literature. In this chapter, we provide a selective overview of MARL, with focus on algorithms backed by theoretical analysis. More specifically, we review the theoretical results of MARL algorithms mainly within two representative frameworks, Markov/stochastic games and extensive-form games, in accordance with the types of tasks they address, i.e., fully cooperative, fully competitive, and a mix of the two. We also introduce several significant but challenging applications of these algorithms. Orthogonal to the existing reviews on MARL, we highlight several new angles and taxonomies of MARL theory, including learning in extensive-form games, decentralized MARL with networked agents, MARL in the mean-field regime, (non-)convergence of policy-based methods for learning in games, etc. Some of the new angles extrapolate from our own research endeavors and interests. Our overall goal with this chapter is, beyond providing an assessment of the current state of the field on the mark, to identify fruitful future research directions on theoretical studies of MARL. We expect this chapter to serve as continuing stimulus for researchers interested in working on this exciting while challenging topic.Recent years have witnessed sensational advances of reinforcement learning (RL) in many prominent sequential decision-making problems, such as playing the game of Go (Silver

DAN: Decentralized Attention-based Neural Network to Solve the MinMax Multiple Traveling Salesman Problem

“…We consider a fully observable world where one agent can access the states of all cities and all agents. Although a partial observation is more common in decentralized MARL [15], a global observation is necessary to make our model comparable to baseline algorithms, and partial observability will be considered in future works. The observation of each agent consist of three parts: the cities state, the agents state, and a global mask.…”

Social physics

“…In MADDPG, the actor is used to select actions, while a central critic evaluates those actions by observing the joint state and actions of all agents. In this sense, MADDPG follows the centralised learning with decentralised execution paradigm [641,642,643], which assumes unrestricted communication bandwidth during training, as well as the central controller's ability to receive and process all agents' information. To relax these assumptions, Flexible Fully-decentralised Approximate Actor-critic (F2A2) algorithm [644] was proposed as a variant of multi-agent reinforcement learning based on decentralised training with decentralised execution.…”

Large-scale Machine Learning Cluster Scheduling via Multi-agent Graph Reinforcement Learning

“…To prevent high variance in Q values and ensure expedited convergence of the policy network, we adopt actor-critic algorithm [18], which introduces an advantage function to replace 𝑄 value for policy gradient calculation, i.e., 𝛿 = 𝑟 + 𝛾𝑉 𝜋 𝜃 (𝑠 ; 𝜔) − 𝑉 𝜋 𝜃 (𝑠; 𝜔), where 𝑉 𝜋 𝜃 (𝑠; 𝜔) is calculated as the expected cumulative reward following the policy 𝜋 𝜃 from state 𝑠, over all possible actions; we use a value network (the critic) with 𝜔 as the set of parameters to estimate 𝑉 𝜋 𝜃 (𝑠; 𝜔). Specifically, the actor is a policy network with input 𝑠, and output 𝜋 𝜃 (𝑠); the critic has input 𝑠, but the output layer is a linear neuron without any activation function.…”

“…Multi-agent Reinforcement Learning. Recently MARL has achieved promising results in various application domains, e.g., traffic engineering, video games, etc [18]. We treat the DL cluster schedulers as a multi-agent system.…”