Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning

Abstract

Reward decomposition is a critical problem in centralized training with decentralized execution (CTDE) paradigm for multi-agent reinforcement learning. To take full advantage of global information, which exploits the states from all agents and the related environment for decomposing Q values into individual credits, we propose a general meta-learning-based Mixing Network with Meta Policy Gradient (MNMPG) framework to distill the global hierarchy for delicate reward decomposition. The excitation signal for learning global hierarchy is deduced from the episode reward difference between before and after "exercise updates" through the utility network. Our method is generally applicable to the CTDE method using a monotonic mixing network. Experiments on the StarCraft II micromanagement benchmark demonstrate that our method just with a simple utility network is able to outperform the current state-of-the-art MARL algorithms on 4 of 5 super hard scenarios. Better performance can be further achieved when combined with a role-based utility network. IntroductionMulti-agent deep reinforcement learning algorithms (MARL) have recently shown extraordinary performance in various games like DOTA2 Berner et al. (2019), StarCraft Samvelyan et al. (2019), and Honor of Kings Ye et al. (2020). The framework of centralized training with decentralized execution (CTDE) Gupta et al. (2017); Rashid et al. (2018), which enjoys the advantages of joint action learning Claus & Boutilier (1998) and independent learning Tan (1993), is one of the popular frameworks for solving collaborative multi-agent tasks. Recent research on the CTDE framework can be divided into two categories: One is to enhance agents' ability with emphasis on individually processing local observations, for instance, allocating a role Wang et al. (2020b,c) or a mode of exploration Mahajan et al. (2019) to each agent, which may need extra prior information and consume more computational resources on decentralized execution due to extensive inference network. Another aims to decompose the single reward to each agent accurately, in other words, training a delicate mixing network for local utility values Rashid et al. (2018); Yang et al. (2020); Wang et al. (2020a), or training a new joint action-value to factorizing tasks Son et al. (2019). The latter's empirical performance on challenging tasks is relatively limited by training instability and demand for delicate parameter adjustment.The current methods on mixing network usually utilize the global information from the full state roughly, such as Rashid et al. ( 2018) which only takes the full state vector as the input of the mixing network. These methods lack an explicit hierarchy, the distilled information from the full state, to decompose Q values into individual credit. Inspired by Meta-DDPG Xu et al. (2018a), where meta-learning is adopted for exploration, in this paper, we present a general meta-learning-based framework called Mixing Network with Meta Policy Gradient (MNMPG) for exploration on better Preprint. Under revi...

Researches advanced in multi-agent credit assignment in reinforcement learning

“…While some researchers [11] pay more attention to coordinated relationship among agents, inventing coordination knowledge transfer that has better generalization and scalability. Besides, Shao et al [40] prefer to the mechanism of self-improvement, which uses no prior information. However, the challenge in the logic criterion of CA is the prior knowledge cannot be presented clearly and applied to reinforcement learning, or it is not always useful for every multi-agent system.…”

“…In this module, the weights of each layer are generated by using a hyper network with absolute value calculations so that the integration of local Q-value satisfies the monotonicity constraint and the use of global information is greater and more flexible to estimate the Q-value of joint actions, improving the learning and convergence of the global Q-value. Shao et al [40] propose Mixing Network with Meta Policy Gradient (MNMPG), which assign proper credit to each agent using a global hierarchy with meta policy gradient. Zhou et al [11] propose a level-adaptive QTransformer (LA-QTransformer) that is a novel mixing network with a multi-head attention module to combine all coordination patterns and generate the credit assignment weights.…”