Learning Explicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning via Polarization Policy Gradient

Chen, Wubing; Li, Wenbin; Liu, Xiao; Yang, Shuyuan

doi:10.48550/arxiv.2210.05367

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multi-Agent Policy Gradient The policy gradient in stochastic MAPG methods has the form (Foerster et al 2018;Wang et al 2020c;Lou et al 2023b;Chen et al 2022), where objective G i varies across different methods, such as counterfactual advantage (Foerster et al 2018) and polarized joint-action value (Chen et al 2022). The objective in DOP is individual aristocratic utility (Wolpert and Tumer 2001), which ignores other agents' utilities to avoid the CDM issue, but the cooperation is also limited by this objective.…”

Section: Related Workmentioning

confidence: 99%

“…The objective in DOP is individual aristocratic utility (Wolpert and Tumer 2001), which ignores other agents' utilities to avoid the CDM issue, but the cooperation is also limited by this objective. It is worth noting that polarized joint-action value (Chen et al 2022) (Zhang et al 2021;Peng et al 2021;Zhou, Lan, and Aggarwal 2022) adopt value factorization to mix individual Q values to get Q π tot . As the global Q value is determined by the centralized critic for all agents, sub-optimal actions of one agent will easily influence all others.…”

Section: Related Workmentioning

confidence: 99%

“…The centralized critic approach has an inherent problem known as centralized-decentralized mismatch (CDM) (Wang et al 2020c;Chen et al 2022). The CDM issue refers to sub-optimal, or explorative actions of some agents negatively affecting policy learning of others, causing catastrophic miscoordination.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient

Lou,

Zhang,

Norman

et al. 2024

AAAI

View full text Add to dashboard Cite

Multi-Agent Policy Gradient (MAPG) has made significant progress in recent years. However, centralized critics in state-of-the-art MAPG methods still face the centralized-decentralized mismatch (CDM) issue, which means sub-optimal actions by some agents will affect other agent's policy learning. While using individual critics for policy updates can avoid this issue, they severely limit cooperation among agents. To address this issue, we propose an agent topology framework, which decides whether other agents should be considered in policy gradient and achieves compromise between facilitating cooperation and alleviating the CDM issue. The agent topology allows agents to use coalition utility as learning objective instead of global utility by centralized critics or local utility by individual critics. To constitute the agent topology, various models are studied. We propose Topology-based multi-Agent Policy gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove the policy improvement theorem for stochastic TAPE and give a theoretical explanation for the improved cooperation among agents. Experiment results on several benchmarks show the agent topology is able to facilitate agent cooperation and alleviate CDM issue respectively to improve performance of TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm are devised to show the efficacy of the agent topology.

show abstract