When Being Selfish Prevails: The Impact of Sociality Regimes on Heterogeneous Cooperative-Competitive Multi-agent Reinforcement Learning

Abstract

Work in multi-agent reinforcement learning (MARL) tends to compare algorithms choosing some particular sociality regime for sharing (or not) the rewards, assuming this choice is optimal. For instance, the popular multi-agent deep deterministic policy gradient algorithm (MADDPG), when evaluated in a predator and prey game, used a different regime when acting as a predator than when acting as a prey. In this paper, we question what kind of sociality regimes (selfish, egalitarian, or altruistic) should be used in multi-agent algorithms. In the past, choices of the sociality regime have been done only for homogeneous teams, and without normalising the effect of the sociality regimes. Here we introduce a normalisation of the weights of the sociality regimes so that the sum of all rewards is independent of the sociality regime. With this, we extend the analysis to heterogeneous teams in cooperative-competitive MARL situations since team composition is very relevant to the sociality choice. We find that the selfish regime is advantageous for both prey and predator teams, and for both homogeneous and heterogeneous teams. The takeaway message is that, unlike common practice in MARL, any study of homogeneous and heterogeneous cooperative-competitive multi-agent reinforcement learning teams should also take into account the sociality regimes before making conclusions on the preference of any algorithm.