Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning

Abstract

Many tasks in artificial intelligence require the collaboration of multiple agents. We exam deep reinforcement learning for multi-agent domains. Recent research efforts often take the form of two seemingly conflicting perspectives, the decentralized perspective, where each agent is supposed to have its own controller; and the centralized perspective, where one assumes there is a larger model controlling all agents. In this regard, we revisit the idea of the master-slave architecture by incorporating both perspectives within one framework. Such a hierarchical structure naturally leverages advantages from one another. The idea of combining both perspectives is intuitive and can be well motivated from many real world systems, however, out of a variety of possible realizations, we highlights three key ingredients, i.e. composed action representation, learnable communication and independent reasoning. With network designs to facilitate these explicitly, our proposal consistently outperforms latest competing methods both in synthetic experiments and when applied to challenging StarCraft 1 micromanagement tasks.

Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration

“…The data are used to recover the policy by imitation learning (2) and the learned policy is further used to calculate Q D (9). As same as most MARL works (Usunier et al 2016;Peng et al 2017;Kong et al 2017;, the opponent's policy can be simulated as a part of environment in learning. It is reasonable because our goal is to learn multi-agent cooperation rather than competition.…”

“…The Nash Equilibrium reduces the exploration state-action space and leads to more effective learning of cooperation. A similar method is proposed in (Lanctot et al 2017) where one agent update it's policy by sampling other agent's policies from their individual meta-strategies. Compared with (Lanctot et al 2017), the proposed method assumes all agents are performing as Nash Equilibrium jointly and it is more reasonable to multi-agent cooperation.…”

“…A similar method is proposed in (Lanctot et al 2017) where one agent update it's policy by sampling other agent's policies from their individual meta-strategies. Compared with (Lanctot et al 2017), the proposed method assumes all agents are performing as Nash Equilibrium jointly and it is more reasonable to multi-agent cooperation. All of these deep MARL methods assume that the reward is available at every state.…”

“…The proposed method are tested in combat RTS game where the task is to coordinate multiple allied agents to defeat their enemies controlled by an opponent in a real-time scenario. The combat game has a high requirement for multiagent cooperation and has been widely utilized to test multiagent based methods (Usunier et al 2016;Peng et al 2017;Kong et al 2017;Lowe et al 2017;Rashid et al 2018;Churchill, Saffidine, and Buro 2012;Churchill and Buro 2013;Lelis 2017) learning point of view, combat game provides a challenging environment to test AI algorithms because its stateaction space is extremely large and the time allowed for planning is on the order of milliseconds. For example, in a specific 15 vs. 15 scenario, the number of points in joint state-action space is almost 10 500 which is much larger than for Go (10 170 ) (Silver et al 2016).…”

Learning Agent Communication under Limited Bandwidth by Message Pruning

“…Deterministic Policy Gradient (DPG) (Silver et al 2014) is a special actor-critic algorithm where the actor adopts a deterministic policy µ θ : S → A and the action space A is continuous. Deep DPG (DDPG) (Lillicrap et al 2015) applies DNN µ θ (s) and Q(s, a; w) to represent the actor and the critic, respectively. DDPG is an off-policy method.…”