Reinforcement Learning for Linear Quadratic Control is Vulnerable Under Cost Manipulation

Abstract

In this work, we study the deception of a Linear-Quadratic-Gaussian (LQG) agent by manipulating the cost signals. We show that a small falsification on the cost parameters will only lead to a bounded change in the optimal policy and the bound is linear on the amount of falsification the attacker can apply on the cost parameters. We propose an attack model where the goal of the attacker is to mislead the agent into learning a 'nefarious' policy with intended falsification on the cost parameters. We formulate the attack's problem as an optimization problem, which is proved to be convex, and developed necessary and sufficient conditions to check the achievability of the attacker's goal.We showcase the adversarial manipulation on two types of LQG learners: the batch RL learner and the other is the adaptive dynamic programming (ADP) learner. Our results demonstrate that with only 2.296% of falsification on the cost data, the attacker misleads the batch RL into learning the 'nefarious' policy that leads the vehicle to a dangerous position. The attacker can also gradually trick the ADP learner into learning the same 'nefarious' policy by consistently feeding the learner a falsified cost signal that stays close to the true cost signal. The aim of the paper is to raise people's awareness of the security threats faced by RL-enabled control systems.

Policy Poisoning in Batch Learning for Linear Quadratic Control Systems via State Manipulation

“…Understanding these attacks and developing countermeasures to guarantee the safety of the systems are critical [12]. Previous works have explored policy poisoning attacks occurring via manipulation of the system's cost measurements in a discrete-time control system [13], [14]. Our work focuses on the case of policy poisoning through sensor data manipulation on a batch-learning agent in the continuous-time setting, which we believe to be a more realistic and applicable attack scenario.…”