Analysing factorizations of action-value networks for cooperative multi-agent reinforcement learning

Abstract

Recent years have seen the application of deep reinforcement learning techniques to cooperative multi-agent systems, with great empirical success. However, given the lack of theoretical insight, it remains unclear what the employed neural networks are learning, or how we should enhance their learning power to address the problems on which they fail. In this work, we empirically investigate the learning power of various network architectures on a series of one-shot games. Despite their simplicity, these games capture many of the crucial problems that arise in the multi-agent setting, such as an exponential number of joint actions or the lack of an explicit coordination mechanism. Our results extend those in Castellini et al. (Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS’19.International Foundation for Autonomous Agents and Multiagent Systems, pp 1862–1864, 2019) and quantify how well various approaches can represent the requisite value functions, and help us identify the reasons that can impede good performance, like sparsity of the values or too tight coordination requirements.

Deep Coordination Graphs

“…If the employed value function does not have the representational capacity to distinguish the values of coordinated and uncoordinated actions, an optimal policy cannot be learned. However, Castellini et al (2019) show that higher-order factorization of the value function works surprisingly well in one-shot games that are vulnerable to relative overgeneralization, even if each factor depends on the actions of only a small subset of agents. Such a higher-order factorization can be expressed as an undirected coordination graph (CG, Guestrin et al, 2002a), where each vertex represents one agent and each (hyper-)edge one payoff function over the joint action space of the connected agents.…”

“…Sparse cooperative Q-learning (Kok & Vlassis, 2006) applies CGs to MARL but does not scale to modern benchmarks, as each payoff function (f 12 and f 23 in Figure 1b) is represented as a table over the state and joint action space of the connected agents. Castellini et al (2019) use neural networks to approximate payoff functions, but only in one-shot games, and still require a unique function for each edge in the CG. Consequently, each Sunehag et al, 2018) corresponds to an unconnected CG.…”

Context-Aware Sparse Deep Coordination Graphs

“…Importantly, we show that carefully learned sparse graphs can significantly outperform complete graphs. We expect these observations to be a good supplementation to Castellini et al [23] and can eliminate any possible misunderstanding about sparse coordination graphs. Stability and Recommendation Table 3 in Appendix A shows the stability of each implementation on all tasks across the MACO benchmark.…”

“…Again, we observe a performance gap between complete coordination graphs and most implementations of context-aware sparse graphs. Castellini et al [23] finds that (randomly) sparse coordination graphs perform much worse than full graphs. This is aligned with our experimental results.…”

“…Zhang and Lesser [22] propose to learn minimized dynamic coordination sets for each agent, but the computational complexity grows exponentially with the neighborhood size of an agent. Recently, Castellini et al [23] study the representational capability of several sparse graphs but focus on random topologies and stateless games. In this paper, we push these previous works further by proposing a novel deep method that learns context-aware sparse coordination graphs adaptive to the dynamic coordination requirements of agents.…”