Certifiably Robust Policy Learning against Adversarial Communication in Multi-agent Systems

Abstract

Communication is important in many multi-agent reinforcement learning (MARL) problems for agents to share information and make good decisions. However, when deploying trained communicative agents in a real-world application where noise and potential attackers exist, the safety of communication-based policies becomes a severe issue that is underexplored. Specifically, if communication messages are manipulated by malicious attackers, agents relying on untrustworthy communication may take unsafe actions that lead to catastrophic consequences. Therefore, it is crucial to ensure that agents will not be misled by corrupted communication, while still benefiting from benign communication. In this work, we consider an environment with N agents, where the attacker may arbitrarily change the communication from any C < N −1 2 agents to a victim agent. For this strong threat model, we propose a certifiable defense by constructing a message-ensemble policy that aggregates multiple randomly ablated message sets. Theoretical analysis shows that this message-ensemble policy can utilize benign communication while being certifiably robust to adversarial communication, regardless of the attacking algorithm. Experiments in multiple environments verify that our defense significantly improves the robustness of trained policies against various types of attacks.

Adversarial attack and defense technologies in natural language processing: A survey

“…These defenses mainly focus on the adversarial perturbations directly applied to the agent's inputs. [61] propose a certifiable defense against adversarial communication in MARL systems. To the best of our knowledge, our paper is the first to provide provable convergence guarantees for adversarial training against adversarial attacks on behaviors of other agents in the environment.…”

Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning

“…RL has also been shown to be susceptible to perturbation in the observations of an RL agent (Huang et al 2017;Behzadan and Munir 2017) or in environments (Gleave et al 2019). Some adversarial defence works for RL are proposed (Donti et al 2020;Eysenbach and Levine 2021;Shen et al 2020;Sun et al 2022) and then towards these defences, stronger attacks are proposed (Salman et al 2019;Russo and Proutiere 2019). To end this repeated game, Wu et al (2021) and Kumar, Levine, and Feizi (2021) proposed to use probabilistic approaches to provide robustness certification for RLs.…”