Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning

Abstract

Deep Reinforcement Learning (DRL) has been a promising solution to many complex decision-making problems. Nevertheless, the notorious weakness in generalization among environments prevent widespread application of DRL agents in real-world scenarios. Although advances have been made recently, most prior works assume sufficient online interaction on training environments, which can be costly in practical cases. To this end, we focus on an offline-training-online-adaptation setting, in which the agent first learns from offline experiences collected in environments with different dynamics and then performs online policy adaptation in environments with new dynamics. In this paper, we propose Policy Adaptation with Decoupled Representations (PAnDR) for fast policy adaptation. In offline training phase, the environment representation and policy representation are learned through contrastive learning and policy recovery, respectively. The representations are further refined by mutual information optimization to make them more decoupled and complete. With learned representations, a Policy-Dynamics Value Function (PDVF) network is trained to approximate the values for different combinations of policies and environments from offline experiences. In online adaptation phase, with the environment context inferred from few experiences collected in new environments, the policy is optimized by gradient ascent with respect to the PDVF. Our experiments show that PAnDR outperforms existing algorithms in several representative policy adaptation problems.

Optimal Attack and Defense for Reinforcement Learning

“…For example, Tessler, Efroni, and Mannor (2019); Lee et al (2021) considered action attacks. Reward poisoning attacks are the focus of the work by Zhang et al (2020b); Rangi et al (2022). In fact, a combination action and reward attack are devised by Rangi et al (2022).…”

“…Reward poisoning attacks are the focus of the work by Zhang et al (2020b); Rangi et al (2022). In fact, a combination action and reward attack are devised by Rangi et al (2022). Most of these works consider the policy teaching setting, where the attacker's goal is for the victim to follow a fixed policy π † .…”

Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning

“…Offline Reward Poisoning: Ma et al (2019); Rakhsha et al (2020Rakhsha et al ( , 2021a; Rangi et al (2022b); Zhang and Parkes (2008); Zhang, Parkes, and Chen (2009) focus on adversarial attack on offline single-agent reinforcement learners. Gleave et al (2019); Guo et al (2021) study the poisoning attack on multi-agent reinforcement learners, assuming that the attacker controls one of the learners.…”

Data Poisoning to Fake a Nash Equilibria for Markov Games

“…Data poisoning attacks have been well studied in supervised learning (intentionally forcing the learner to train a wrong classifier) and reinforcement learning (wrong policy) (Banihashem et al 2022;Huang and Zhu 2019;Liu and Lai 2021;Rakhsha et al 2021aRakhsha et al ,b, 2020Sun, Huo, and Huang 2020;Zhang et al 2020;Ma et al 2019;Rangi et al 2022;Zhang and Parkes 2008;Zhang, Parkes, and Chen 2009). Can data poisoning attacks be a threat to Markov Games, too?…”