Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment

Abstract

Extending transfer learning to cooperative multi-agent reinforcement learning (MARL) has recently received much attention. In contrast to the single-agent setting, the coordination indispensable in cooperative MARL constrains each agent's policy. However, existing transfer methods focus exclusively on agent policy and ignores coordination knowledge. We propose a new architecture that realizes robust coordination knowledge transfer through appropriate decomposition of the overall coordination into several coordination patterns. We use a novel mixing network named level-adaptive QTransformer (LA-QTransformer) to realize agent coordination that considers credit assignment, with appropriate coordination patterns for different agents realized by a novel level-adaptive Transformer (LA-Transformer) dedicated to the transfer of coordination knowledge. In addition, we use a novel agent network named Population Invariant agent with Transformer (PIT) to realize the coordination transfer in more varieties of scenarios. Extensive experiments in StarCraft II micro-management show that LA-QTransformer together with PIT achieves superior performance compared with state-of-the-art baselines.

Researches advanced in multi-agent credit assignment in reinforcement learning

“…Most researches of MACA like Harati et al [13] use agent-level knowledge, which is based on different characteristics of each agent to distribute rewards. While some researchers [11] pay more attention to coordinated relationship among agents, inventing coordination knowledge transfer that has better generalization and scalability. Besides, Shao et al [40] prefer to the mechanism of self-improvement, which uses no prior information.…”

“…Shao et al [40] propose Mixing Network with Meta Policy Gradient (MNMPG), which assign proper credit to each agent using a global hierarchy with meta policy gradient. Zhou et al [11] propose a level-adaptive QTransformer (LA-QTransformer) that is a novel mixing network with a multi-head attention module to combine all coordination patterns and generate the credit assignment weights.…”

“…Population invariant agent via Transformer (PIT) is designed by Zhou et al [11] to achieve coordination transfer in more general scenarios, which consists of three parts, the Property Group Module, the GRU Core, and the Adaptive Action Module.…”

“…(3) Mixing network. In contrast to the single-agent setting, policies about each agent are constrained by the coordination indispensable in MARL [11]. However, many algorithms in MARL ignore the coordinated relationship.…”

Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning

“…On the other hand, MAAC [14] introduces the attention mechanism to learn the interaction relationships, while ATOC [15] exploits attentional communication to make cooperative decisions. UPDeT [37] and PIT [38] utilize attention-based semantic alignment between the input entities and the output actions to decompose the policy. Despite the promising results achieved, these works mainly rely on dense attention and have to focus on the irrelevance entities.…”