Greedy-based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning

Abstract

Due to the representation limitation of the joint Q value function, multi-agent reinforcement learning (MARL) methods with linear or monotonic value decomposition suffer from the relative overgeneralization. As a result, they can not ensure the optimal coordination. Existing methods address the relative overgeneralization by achieving complete expressiveness or learning a bias, which is insufficient to solve the problem. In this paper, we propose the optimal consistency, a criterion to evaluate the optimality of coordination. To achieve the optimal consistency, we introduce the True-Global-Max (TGM) principle for linear and monotonic value decomposition, where the TGM principle can be ensured when the optimal stable point is the unique stable point. Therefore, we propose the greedy-based value representation (GVR) to ensure the optimal stable point via inferior target shaping and eliminate the non-optimal stable points via superior experience replay. Theoretical proofs and empirical results demonstrate that our method can ensure the optimal consistency under sufficient exploration. In experiments on various benchmarks, GVR significantly outperforms state-of-the-art baselines.

Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning

“…To address this problem, QPLEX [25] and QTRAN [22] aim to learn value functions with complete expressiveness capacity. However, reports are that they perform poorly when being used in practice [5,24]. This is because learning the complete expressiveness is impractical in complicated MARL tasks due to the challenging exploration in large joint action spaces.…”

“…6.4.1 Comparison among Using Different 𝑧 Dimensions. We compare the performance of AVGM using different 𝑧 dimensions (8,16,24,32,40) in pursuit. The results are shown in Figure 5.…”