Online Robust Policy Learning in the Presence of Unknown Adversaries

Abstract

The growing prospect of deep reinforcement learning (DRL) being used in cyber-physical systems has raised concerns around safety and robustness of autonomous agents. Recent work on generating adversarial attacks have shown that it is computationally feasible for a bad actor to fool a DRL policy into behaving sub optimally. Although certain adversarial attacks with specific attack models have been addressed, most studies are only interested in off-line optimization in the data space (e.g., example fitting, distillation). This paper introduces a Meta-Learned Advantage Hierarchy (MLAH) framework that is attack model-agnostic and more suited to reinforcement learning, via handling the attacks in the decision space (as opposed to data space) and directly mitigating learned bias introduced by the adversary. In MLAH, we learn separate sub-policies (nominal and adversarial) in an online manner, as guided by a supervisory master agent that detects the presence of the adversary by leveraging the advantage function for the sub-policies. We demonstrate that the proposed algorithm enables policy learning with significantly lower bias as compared to the state-of-the-art policy learning approaches even in the presence of heavy state information attacks. We present algorithm analysis and simulation results using popular OpenAI Gym environments.

The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning

“…The results indicate that in both environments, DQN agents are able to recover from noncontiguous attacks with attack probabilities p = 0.2 and p = 0.4 and converge to optimal performance, while they fail to recover under attacks with p = 0.8 and p = 1.0 (contiguous attack). It is observed that for the agents that Hierarchical RL Havens et al [71] Game Theoretic Ogunmolu et al [72], Bravo & Mertikopoulos [73] recover, the training performance deteriorates almost uniformly until a minimum point is reached, from which onward the agent begins to recover and adjust the policy towards optimal performance. The authors' interpretation of this behavior is based on the statistics of experience replay: for the agent to recover from adversarial perturbations, the number of interactions with the perturbed environment must reach a critical threshold, so that the randomly sampled batches from the experience memory can represent the statistical significance of perturbations.…”

“…During policy learning, information perturbation can be generally viewed as a bias that can prevent the agent from effectively learning the desired policy. Inspired by this idea, Havens et al [71] propose a hierarchical metalearning framework, named Meta-Learned Advantage Hierarchy (MLAH). Their work considers a policy learning problem where there are periods of adversarial attacks that corrupt state observations during the continuous learning of the agent, and aims at the online mitigation of the bias introduced by the attack into the nominal policy.…”