Learning Transferable Cooperative Behavior in Multi-Agent Teams

Agarwal, Akshat; Kumar, Sandeep; Sycara, Katia

doi:10.48550/arxiv.1906.01202

Cited by 12 publications

(28 citation statements)

References 13 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Methods for Dynamic Team Compositions Several recent works in transfer learning and curriculum learning attempt to transfer policy of small teams for larger teams (Carion et al, 2019;Shu & Tian, 2019;Agarwal et al, 2019;Wang et al, 2020b;Long et al, 2020). These works mostly consider teams with different numbers of homogeneous agents.…”

Section: Centralized Training With Decentralized Executionmentioning

confidence: 99%

Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition

Liu,

Stone

et al. 2021

Preprint

View full text Add to dashboard Cite

In real-world multiagent systems, agents with different capabilities may join or leave without altering the team's overarching goals. Coordinating teams with such dynamic composition is challenging: the optimal team strategy varies with the composition. We propose COPA, a coach-player framework to tackle this problem. We assume the coach has a global view of the environment and coordinates the players, who only have partial views, by distributing individual strategies. Specifically, we 1) adopt the attention mechanism for both the coach and the players; 2) propose a variational objective to regularize learning; and 3) design an adaptive communication method to let the coach decide when to communicate with the players. We validate our methods on a resource collection task, a rescue game, and the StarCraft micromanagement tasks. We demonstrate zeroshot generalization to new team compositions. Our method achieves comparable or better performance than the setting where all players have a full view of the environment. Moreover, we see that the performance remains high even when the coach communicates as little as 13% of the time using the adaptive communication strategy.

show abstract

Section: Centralized Training With Decentralized Executionmentioning

confidence: 99%

Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition

Liu,

Stone

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Multi-Actor-Attention-Critic (MAAC) is proposed in [19] to aggregate information using attention mechanism from all the other agents. Similarly, [11], [13], [20] also employ the attention mechanism to learn a representation for the action-value function. However, the communication graphs used there are either dense or ad-hoc (k nearest neighbors), which makes the learning difficult.…”

Section: A Related Workmentioning

confidence: 99%

“…Graph neural network (GNN) is a specific neural network architecture in which permutation-invariance features can be embedded via graph pooling operations, so this approach has been applied in MARL [11]- [13] to exploit the interchangeability. As MARL is a non-structural scenario where the links/connections between the nodes/agents are ambiguous to decide, a graph has to be created in advance to apply GNN for MARL.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph

Shen

How

2020

2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

The complexity of multiagent reinforcement learning (MARL) in multiagent systems increases exponentially with respect to the agent number. This scalability issue prevents MARL from being applied in large-scale multiagent systems. However, one critical feature in MARL that is often neglected is that the interactions between agents are quite sparse. Without exploiting this sparsity structure, existing works aggregate information from all of the agents and thus have a high sample complexity. To address this issue, we propose an adaptive sparse attention mechanism by generalizing a sparsity-inducing activation function. Then a sparse communication graph in MARL is learned by graph neural networks based on this new attention mechanism. Through this sparsity structure, the agents can communicate in an effective as well as efficient way via only selectively attending to agents that matter the most and thus the scale of the MARL problem is reduced with little optimality compromised. Comparative results show that our algorithm can learn an interpretable sparse structure and outperforms previous works by a significant margin on applications involving a large-scale multiagent system.

show abstract

“…Existing CTDE research covers important topics such as division of agents [27], diversification [32] and exploration [19]. Recent works [29,11,1,17,16] have also started to make progress in transfer learning in cooperative MARL. For example, Liu et al [16] use policy distillation [22] to achieve fixed agent transfer learning.…”

Section: Introductionmentioning

confidence: 99%

Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment

Zhou¹,

Zhang²,

Shao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Extending transfer learning to cooperative multi-agent reinforcement learning (MARL) has recently received much attention. In contrast to the single-agent setting, the coordination indispensable in cooperative MARL constrains each agent's policy. However, existing transfer methods focus exclusively on agent policy and ignores coordination knowledge. We propose a new architecture that realizes robust coordination knowledge transfer through appropriate decomposition of the overall coordination into several coordination patterns. We use a novel mixing network named level-adaptive QTransformer (LA-QTransformer) to realize agent coordination that considers credit assignment, with appropriate coordination patterns for different agents realized by a novel level-adaptive Transformer (LA-Transformer) dedicated to the transfer of coordination knowledge. In addition, we use a novel agent network named Population Invariant agent with Transformer (PIT) to realize the coordination transfer in more varieties of scenarios. Extensive experiments in StarCraft II micro-management show that LA-QTransformer together with PIT achieves superior performance compared with state-of-the-art baselines.

show abstract

Learning Transferable Cooperative Behavior in Multi-Agent Teams

Cited by 12 publications

References 13 publications

Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition

Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition

Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph

Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment

Contact Info

Product

Resources

About