Tactics of Adversarial Attack on Deep Reinforcement Learning Agents

Abstract

We introduce two tactics, namely the strategicallytimed attack and the enchanting attack, to attack reinforcement learning agents trained by deep reinforcement learning algorithms using adversarial examples. In the strategically-timed attack, the adversary aims at minimizing the agent's reward by only attacking the agent at a small subset of time steps in an episode. Limiting the attack activity to this subset helps prevent detection of the attack by the agent. We propose a novel method to determine when an adversarial example should be crafted and applied. In the enchanting attack, the adversary aims at luring the agent to a designated target state. This is achieved by combining a generative model and a planning algorithm: while the generative model predicts the future states, the planning algorithm generates a preferred sequence of actions for luring the agent. A sequence of adversarial examples is then crafted to lure the agent to take the preferred sequence of actions. We apply the proposed tactics to the agents trained by the state-of-the-art deep reinforcement learning algorithm including DQN and A3C. In 5 Atari games, our strategically-timed attack reduces as much reward as the uniform attack (i.e., attacking at every time step) does by attacking the agent 4 times less often. Our enchanting attack lures the agent toward designated target states with a more than 70% success rate. Example videos are available at http: //yclin.me/adversarial_attack_RL/.

Minimalistic Attacks: How Little It Takes to Fool Deep Reinforcement Learning Policies

“…adversarial examples that change almost every pixel in the input state) has previously been generated by a white-box policy access based approach [21], where the adversarial examples are computed via backpropagation. Lin et al [4] proposed the strategically-timed attack and the so-called enchanting attack, but the adversary generation is still based on a white-box policy access assumption and full-state perturbation. Besides, Kos et al [22] compared the influence of full-state perturbations with random noise, and utilized the value function to guide the adversary injection.…”

“…These adversaries can easily fool even seemingly high performing deep learning models with human imperceptible perturbations. Such vulnerabilities of deep learning models have been well studied in supervised learning, and also to some extent in RL [3], [4], [20].…”

“…To solve the optimization problem formulated in Eq. (2), many white-box based approaches (e.g., FSGM [18]) have been applied in previous efforts [3], [4]. These white-box methods are essentially gradient-based methods, as they generate adversarial examples by back-propagating through the policy network to calculate the gradient of a cost function with respect to the input state, i.e., ∇ st J(π, θ, ∆a t , s t ).…”

“…The exact choice of discrepancy measure D(·) is expected to have a significant impact on the attack performance, as different measures shall capture different patterns of similarities. Previous works [3], [4], [20] apply the Euclidean norm (e.g., L 1 , L 2 , L ∞ ) between π(·|s t + δ t ) and π(·|s t ). However, such L p norm cannot guarantee a successful untargeted attack, since maximizing the L p norm does not ensure that the action selection for the perturbed state has been altered.…”

Adversarial manipulation of human decision-making

“…One line of systematic study of the first question started in image classification, with seminal early observations from Szegedy et al (Szegedy et al, 2013) that deep artificial neural networks are brittle to adversarial change in inputs that would otherwise be imperceptible to the human eye. This computer vision weakness of the machine has been an angle of attack to design adversaries for reinforcement-learning agents (Lin et al, 2017), followed by general formal insights on adversarial reinforcement learning on the more classical bandit settings (Jun, Li, Ma, & Zhu, 2018). To analyse human choice frailty, our framework involves two steps, the key one also involving a machine-vs-machine adversarial step in which a (deep) reinforcement-learning agent is trained to be an adversary to an RNN; this latter model is trained in a previous step to emulate human decisions following (Dezfouli et al, 2018;Dezfouli, Ashtiani, et al, 2019;Dezfouli, Griffiths, et al, 2019).…”