Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning

Abstract

Recent studies in multi-agent communicative reinforcement learning (MACRL) demonstrate that multi-agent coordination can be significantly improved when communication between agents is allowed. Meanwhile, advances in adversarial machine learning (ML) have shown that ML and reinforcement learning (RL) models are vulnerable to a variety of attacks that significantly degrade the performance of learned behaviours. However, despite the obvious and growing importance, the combination of adversarial ML and MACRL remains largely uninvestigated. In this paper, we make the first step towards conducting message attacks on MACRL methods. In our formulation, one agent in the cooperating group is taken over by an adversary and can send malicious messages to disrupt a deployed MACRLbased coordinated strategy during the deployment phase. We further our study by developing a defence method via message reconstruction. Finally, we address the resulting arms race, i.e., we consider the ability of the malicious agent to adapt to the changing and improving defensive communicative policies of the benign agents. Specifically, we model the adversarial MACRL problem as a two-player zero-sum game and then utilize Policy-Space Response Oracle to achieve communication robustness. Empirically, we demonstrate that MACRL methods are vulnerable to message attacks while our defence method the game-theoretic framework can effectively improve the robustness of MACRL. * Equal contribution.Preprint. Under review.

Multi-Agent Incentive Communication via Decentralized Teammate Modeling

“…Recently, communication in MARL has attracted widespread attention (Giles and Jim 2002;Foerster et al 2016;Hernandez-Leal, Kartal, and Taylor 2019;Lazaridou and Baroni 2020;Xue et al 2021;. DAIL (Foerster et al 2016) is a simple communication mechanism where agents broadcast messages to all teammates that allow the gradient to flow among agents for end-to-end training with reinforcement learning.…”

Certifiably Robust Policy Learning against Adversarial Communication in Multi-agent Systems

“…Communication is important for agents to obtain high rewards, but it can be a double-edged swordit benefits decision making but may make agents vulnerable to perturbations of messages. Communication attacks in MARL has recently attracted increasing attention [3,45,47] with different focuses, as summarized in Section 5. In this paper, we consider a practical and strong threat model where malicious attackers can arbitrarily perturb a subset of communication messages during test time.…”

“…Tu et al [45] investigate the vulnerability of multi-agent autonomous systems against communication attacks, with a focus on vision tasks. Xue et al [47] propose an algorithm to defend against one adversarial communication message by an anomaly detector and a message reconstructor, which are trained with groundtruth labels and messages.…”

“…But they focus on perturbations with small p distances, which do not cover many attack scenarios as discussed in Section 3. Some recent works [3,47,25] take the first step to investigate adversarial communications in MARL and propose several defending methods, such as a learned message filter [47]. However, these empirical defenses do not fully address the aforementioned challenges, and are not guaranteed to be robust, especially under adaptive attacks.…”

Intelligent Blockchain-based Edge Computing via Deep Reinforcement Learning: Solutions and Challenges

“…Moreover, the critic updates the behavior Q-function for the state-action pair of the actor network by minimizing the loss function with the inputs including both local agent's observation and the observations of all other agents. The cooperation of multi-agents helps improve robustness against malicious attacks since this allows multiple agents to monitor the shared policy update via observations [15].…”