Resilient Multi-Agent Reinforcement Learning with Adversarial Value Decomposition

Phan, Thomy; Belzner, Lenz; Gabor, Thomas; Sedlmeier, Andreas; Ritz, Fabian; Linnhoff‐Popien, Claudia

doi:10.1609/aaai.v35i13.17348

Cited by 8 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Background and Related Workmentioning

confidence: 99%

“…Phan et al 62 and Lin et al 21 argued that the higher the number of agents, the higher the potential for failure as each agent can succumb to hardware/software failure or adversarial attacks. To avoid this Phan et al 62 proposed antagonist-ratio training scheme (ARTS) to address both learning and resilience in cooperative multiagent systems (MAS). In ARTS, the cooperation and competitiveness of MAS system are clubbed, where the cooperative (protagonist) MAS learns how to become resilient while achieving the target, whereas the competitive (antagonist) MAS trains to learn possible failure situations, which is later used in testing.…”

Section: Background and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A simulation and experimentation architecture for resilient cooperative multiagent reinforcement learning models operating in contested and dynamic environments

Honhaga,

Szabo

2024

SIMULATION

View full text Add to dashboard Cite

Cooperative multiagent reinforcement learning approaches are increasingly being used to make decisions in contested and dynamic environments, which tend to be wildly different from the environments used to train them. As such, there is a need for a more in-depth understanding of their resilience and robustness in conditions such as network partitions, node failures, or attacks. In this article, we propose a modeling and simulation framework that explores the resilience of four c-MARL models when faced with different types of attacks, and the impact that training with different perturbations has on the effectiveness of these attacks. We show that c-MARL approaches are highly vulnerable to perturbations of observation, action reward, and communication, showing more than 80% drop in the performance from the baseline. We also show that appropriate training with perturbations can dramatically improve performance in some cases, however, can also result in overfitting, making the models less resilient against other attacks. This is a first step toward a more in-depth understanding of the resilience c-MARL models and the effect that contested environments can have on their behavior and toward resilience of complex systems in general.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

A simulation and experimentation architecture for resilient cooperative multiagent reinforcement learning models operating in contested and dynamic environments

Honhaga,

Szabo

2024

SIMULATION

View full text Add to dashboard Cite

show abstract

“…For the observation perturbation of CMARL, Lin et al (2020) learn an adversarial observation policy to attack the system, showing that the ego-system is highly vulnerable to observational perturbations. RADAR (Phan et al 2021) learns resilient MARL policy via adversarial value decomposition. Hu and Zhang (2022) further design an action regularizer to attack the CMARL system efficiently.…”

Section: Related Workmentioning

confidence: 99%

Robust Multi-Agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers

Yuan,

Zhang,

Xue

et al. 2023

AAAI

View full text Add to dashboard Cite

Cooperative Multi-agent Reinforcement Learning (CMARL) has shown to be promising for many real-world applications. Previous works mainly focus on improving coordination ability via solving MARL-specific challenges (e.g., non-stationarity, credit assignment, scalability), but ignore the policy perturbation issue when testing in a different environment. This issue hasn't been considered in problem formulation or efficient algorithm design. To address this issue, we firstly model the problem as a Limited Policy Adversary Dec-POMDP (LPA-Dec-POMDP), where some coordinators from a team might accidentally and unpredictably encounter a limited number of malicious action attacks, but the regular coordinators still strive for the intended goal. Then, we propose Robust Multi-Agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers (ROMANCE), which enables the trained policy to encounter diversified and strong auxiliary adversarial attacks during training, thus achieving high robustness under various policy perturbations. Concretely, to avoid the ego-system overfitting to a specific attacker, we maintain a set of attackers, which is optimized to guarantee the attackers high attacking quality and behavior diversity. The goal of quality is to minimize the ego-system coordination effect, and a novel diversity regularizer based on sparse action is applied to diversify the behaviors among attackers. The ego-system is then paired with a population of attackers selected from the maintained attacker set, and alternately trained against the constantly evolving attackers. Extensive experiments on multiple scenarios from SMAC indicate our ROMANCE provides comparable or better robustness and generalization ability than other baselines.

show abstract

“…Consider the observation perturbation, [16] learns an adversarial observation policy to attack one participant in a cooperative MARL system, demonstrating the high vulnerability of cooperative MARL facing observation perturbation. For the action robustness in cooperative MARL, ARTS [45] and RADAR [46] learn resilient MARL policies via adversarial value decomposition. [17] further designs an action regularizer to attack the cooperative MARL system efficiently.…”

Section: Related Workmentioning

confidence: 99%

Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary generation

Yuan,

Chen,

Zhang

et al. 2023

Front. Comput. Sci.

View full text Add to dashboard Cite

Communication can promote coordination in cooperative Multi-Agent Reinforcement Learning (MARL). Nowadays, existing works mainly focus on improving the communication efficiency of agents, neglecting that real-world communication is much more challenging as there may exist noise or potential attackers. Thus the robustness of the communication-based policies becomes an emergent and severe issue that needs more exploration. In this paper, we posit that the ego system1) trained with auxiliary adversaries may handle this limitation and propose an adaptable method of Multi-Agent Auxiliary Adversaries Generation for robust Communication, dubbed MA3C, to obtain a robust communication-based policy. In specific, we introduce a novel message-attacking approach that models the learning of the auxiliary attacker as a cooperative problem under a shared goal to minimize the coordination ability of the ego system, with which every information channel may suffer from distinct message attacks. Furthermore, as naive adversarial training may impede the generalization ability of the ego system, we design an attacker population generation approach based on evolutionary learning. Finally, the ego system is paired with an attacker population and then alternatively trained against the continuously evolving attackers to improve its robustness, meaning that both the ego system and the attackers are adaptable. Extensive experiments on multiple benchmarks indicate that our proposed MA3C provides comparable or better robustness and generalization ability than other baselines.

show abstract

Resilient Multi-Agent Reinforcement Learning with Adversarial Value Decomposition

Cited by 8 publications

References 22 publications

A simulation and experimentation architecture for resilient cooperative multiagent reinforcement learning models operating in contested and dynamic environments

A simulation and experimentation architecture for resilient cooperative multiagent reinforcement learning models operating in contested and dynamic environments

Robust Multi-Agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers

Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary generation

Contact Info

Product

Resources

About