TrojDRL: Evaluation of Backdoor Attacks on Deep Reinforcement Learning

Abstract

We present TrojDRL, a tool for exploring and evaluating backdoor attacks on deep reinforcement learning agents. TrojDRL exploits the sequential nature of deep reinforcement learning (DRL) and considers different gradations of threat models. We show that untargeted attacks on state-of-the-art actor-critic algorithms can circumvent existing defenses built on the assumption of backdoors being targeted. We evaluated TrojDRL on a broad set of DRL benchmarks and showed that the attacks require only poisoning as little as 0.025% of training data. Compared with existing works of backdoor attacks on classification models, TrojDRL provides a first step towards understanding the vulnerability of DRL agents.

Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers

“…Unlike classification, in DRL, data is generated by an agent interacting with the environment. TrojDRL (Kiourti et al, 2020) proposes data poisoning attacks on DRL agents by altering the observations, after they have been generated by the environment, under a man-in-the-middle (MITM) attack model. More precisely, the observations are altered by a third party, before arriving at the agent to be processed.…”

“…In this work, we consider an attack model where the DRL environments generate poisoned observations. Our contributions are two-fold: 1) Our proposed method of training agents with triggered behavior is simpler than the algorithms outlined by Kiourti et al (2020), cast as a multitask learning problem, which is an approach that, to the best of our knowledge, has not been explored by other published methods for this purpose, and 2) It enables further research into triggers which may not be as easily supported by the MITM attack model, such as triggers that may emerge from multiple agents interacting in the environment.…”

“…Some preliminary research into backdoor attacks for text classification networks also exist (Dai et al, 2019;Sun, 2020). Kiourti et al (2020) were among the first to show that DRL agents are also susceptible to poisoning attacks, and outline an algorithm to poison DRL agents under a man-in-the-middle attack model assumption. In this paper, we expand upon Kiourti et al (2020) in the following ways: 1) We demonstrate how to poison deep reinforcement learning (DRL) agents under an alternate attack model, which motivates an important distinction between simple and in-distribution triggers that impacts vulnerability and defense research.…”

“…Kiourti et al (2020) were among the first to show that DRL agents are also susceptible to poisoning attacks, and outline an algorithm to poison DRL agents under a man-in-the-middle attack model assumption. In this paper, we expand upon Kiourti et al (2020) in the following ways: 1) We demonstrate how to poison deep reinforcement learning (DRL) agents under an alternate attack model, which motivates an important distinction between simple and in-distribution triggers that impacts vulnerability and defense research. 2) We outline and demonstrate a procedure for embedding backdoors into RL agents through the lens of multitask learning, which is interesting due to its simplicity and its connection to another research area in DRL; and provide concrete implementations in popular DRL environments 1 .…”

Backdoor Attacks on Image Classification Models in Deep Neural Networks

“…There are backdoor attacks against other tasks or paradigms, such as Refs. [20][21][22] in the area of natural language processing, Refs. [23,24]in reinforcement learning, Refs.…”