DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning

Abstract

In recent years, multi-agent reinforcement learning (MARL) has presented impressive performance in various applications. However, physical limitations, budget restrictions, and many other factors usually impose constraints on a multi-agent system (MAS), which cannot be handled by traditional MARL frameworks. Specifically, this paper focuses on constrained MASes where agents work cooperatively to maximize the expected team-average return under various constraints on expected team-average costs, and develops a constrained cooperative MARL framework, named DeCOM, for such MASes. In particular, DeCOM decomposes the policy of each agent into two modules, which empowers information sharing among agents to achieve better cooperation. In addition, with such modularization, the training algorithm of DeCOM separates the original constrained optimization into an unconstrained optimization on reward and a constraints satisfaction problem on costs. DeCOM then iteratively solves these problems in a computationally efficient manner, which makes DeCOM highly scalable. We also provide theoretical guarantees on the convergence of DeCOM's policy update algorithm. Finally, we conduct extensive experiments to show the effectiveness of DeCOM with various types of costs in both moderate-scale and large-scale (with 500 agents) environments that originate from real-world applications.

Macro-action-based multi-agent/robot deep reinforcement learning under partial observability

“…Recently, centralized training for decentralized execution (CTDE) [29,30] paradigm attracts the most attention in MARL community, as it handles offline training by utilizing global information while allowing online execution in a decentralized way based on only local information. Current MARL methods mainly implement CTDE in two patterns: one is based on the actor-critic framework that learns a centralized critic to calculate each actor's policy gradients [9,10,12,17,18,31,32,33,34]; the other relies on factorizing centralized Q-value to each decentralized Q-value via networks with variant types of constraints [11,14,15,19,35,36]. Despite these methods having generated remarkable 1.2.…”

“…Independent Actor with Centralized Critic (IACC) is another widespread exploitation of the CTDE paradigm. The state-of-the-art MARL policy gradient approaches [9,10,12,17,18,31,32,33,34] utilize IACC in variant ways and have achieved significant successes 22 CHAPTER 2. BACKGROUND in solving many challenging multi-agent tasks.…”

“…Apart from these positive effects of using a centralized critic, it certainly introduces extra variance in each agent's decentralized policy gradient estimation depending on other agents' actions [72,18]. Therefore, we consider the version of IACC with a joint history-value function as the critic to reduce the variance, and the decentralized policy gradient can be rewritten as:…”

“…Recently, by leveraging deep neural networks to deal with large state and observation input, deep MARL has attracted great attention and achieved many successes in solving challenging multi-agent problems. Unfortunately, the state-of-the-art deep MARL methods [9,10,11,12,13,14,15,16,17,18,19] are not truly applicable to solve large and long-horizon multi-agent tasks in the real-world, because they are originally developed for cases where agents synchronously execute primitive-actions at every time step.…”

“…In recent years, multi-agent policy gradient methods using the actor-critic framework have achieved impressive success in solving a variety of cooperative and competitive domains [9,10,12,17,18,31,32,33,34,84,85,86,87]. However, as these methods assume synchronized primitive-action execution over agents, they struggle to solve large-scale real-world multi-agent problems that involve long-term reasoning and asynchronous behavior.…”