Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks

Abstract

Abstract. Deep learning classifiers are known to be inherently vulnerable to manipulation by intentionally perturbed inputs, named adversarial examples. In this work, we establish that reinforcement learning techniques based on Deep Q-Networks (DQNs) are also vulnerable to adversarial input perturbations, and verify the transferability of adversarial examples across different DQN models. Furthermore, we present a novel class of attacks based on this vulnerability that enable policy manipulation and induction in the learning process of DQNs. We propose an attack mechanism that exploits the transferability of adversarial examples to implement policy induction attacks on DQNs, and demonstrate its efficacy and impact through experimental study of a game-learning scenario.

Reinforcement Learning for Autonomous Defence in Software-Defined Networking

“…Behzadan & Munir [15] establish that adversaries can interfere with the training process of DQNs, preventing the victim from learning the correct policy. Specifically, the attacker applies minimum perturbation to the state observed by the target, so that a different action is chosen as the optimal action at the next state.…”

Mitigation of Policy Manipulation Attacks on Deep Q-Networks with Parameter-Space Noise

“…DQN is trained through optimizing the loss function of equation 4 by SGD. Behzadan and Munir [3] demonstrated that the function approximators of DQN are also vulnerable to adversarial example attacks. In other words, the set of all possible inputs to the approximated functionQ contains elements which cause the approximated functions to generate outputs that are different from the output of the original Q function.…”

“…Accordingly, Behzadan and Munir [3] divide this attack into the two phases of initialization and exploitation. The initialization phase implements processes that must be performed before the target begins interacting with the environment, which are: The exploitation phase implements the attack process and crafting adversarial inputs, such that the target DQN performs an action dictated by π * adv .…”

“…While the interest in deep RL solutions is extending into numerous domains such as intelligent transportation systems [1], finance [6] and critical infrastructure [15], ensuring the security and reliability of such solutions in adversarial conditions is only at its preliminary stages. Recently, Behzadan and Munir [3] reported the vulnerability of deep reinforcement learning algorithms to both test-time and training-time attacks using adversarial examples [9]. This work was followed by a number of further investigations (e.g., [11], [12]), verifying the fragility of deep RL agents to such attacks.…”

“…To this end, we evaluate the performance of Deep Q-Network (DQN) models trained with parameter noise, against the test-time and training-time adversarial example attacks introduced in [3]. Main contributions of this work are: The remainder of this paper is organized as follows: Section 1 reviews the relevant background of DQN, parameter noise training via the NoisyNet approach, and adversarial examples.…”