Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL

Abstract

Evaluating the worst-case performance of a reinforcement learning (RL) agent under the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents. However, finding the optimal adversary is challenging, in terms of both whether we can find the optimal attack and how efficiently we can find it. Existing works on adversarial RL either use heuristics-based methods that may not find the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can find the optimal adversary but may become intractable in a large state space. In this paper, we propose a novel attacking algorithm which has an RL-based "director" searching for the optimal policy perturbation, and an "actor" crafting state perturbations following the directions from the director (i.e. the actor executes targeted attacks). Our proposed algorithm, PA-AD, is theoretically optimal against an RL agent and significantly improves the efficiency compared with prior RL-based works in environments with large or pixel state spaces. Empirical results show that our proposed PA-AD universally outperforms state-of-the-art attacking methods in a wide range of environments. Our method can be easily applied to any RL algorithms to evaluate and improve their robustness.Preprint. Under review.

Certifiably Robust Policy Learning against Adversarial Communication in Multi-agent Systems

“…Adversarial Training does not improve agent's robust performance. As we observed in Figure 2, adversarial training (AT) does not improve the robustness of the agent in both discrete and continuous action space, although AT achieves good robustness againt p attacks in vision tasks [23,48], and against observation attacks [49,40] and action attacks [32] in RL. We hypothesize that it is due to (1) the large number of total agents, (2) the uncertainty of adversarial message channels, and (3) the relatively large perturbation length.…”

“…We hypothesize that it is due to (1) the large number of total agents, (2) the uncertainty of adversarial message channels, and (3) the relatively large perturbation length. To be more specific, in related works [32,49,40], an agent and an attacker are alternately trained, during which the agent learns to adapt to the learned attacker. However, in the threat model we consider, there are C out of N messages significantly perturbed, which it is hard for the agent to adapt to attacks during alternate training.…”

“…Adversarial Robustness of RL Agents Section 3 introduces several adversarial attacks on singleagent and multi-agent problems. To improve the robustness of agents, adversarial training (i.e., introducing adversarial agents to the system during training [32,31,49,40]) and network regularization [50,38,28] are empirically shown to be effective under p attacks, although such robustness is not theoretically guaranteed. Certifying the robustness of a network is an important research problem [33,6,12].…”

“…The learned attack can strategically mislead the victim. As shown in prior works [50,49,40], the theoretically optimal attack (which minimizes the victim's reward) can be formed as an RL problem and learned by RL algorithms. Therefore, we can regard this attack as a worst-case attack for the victim agents.…”

Reinforcement Learning for Feedback-Enabled Cyber Resilience

“…Hence, it is imperative to study RL under an adversarial environment. In the last five years, there is a surge in terms of the number of papers that studies the security issues of RL [104,105,106,107,108,109,110,111,112,113,114,115,116,117]. The vulnerabilities of RL comes from the information exchange between the agent and the environment.…”