On the Robustness of Safe Reinforcement Learning under Observational Perturbations

Abstract

Safe reinforcement learning (RL) trains a policy to maximize the task reward while satisfying safety constraints. While prior works focus on the performance optimality, we find that the optimal solutions of many safe RL problems are not robust and safe against carefully designed observational perturbations. We formally analyze the unique properties of designing effective state adversarial attackers in the safe RL setting. We show that baseline adversarial attack techniques for standard RL tasks are not always effective for safe RL and proposed two new approaches -one maximizes the cost and the other maximizes the reward. One interesting and counter-intuitive finding is that the maximum reward attack is strong, as it can both induce unsafe behaviors and make the attack stealthy by maintaining the reward. We further propose a more effective adversarial training framework for safe RL and evaluate it via comprehensive experiments 1 . This work sheds light on the inherited connection between observational robustness and safety in RL and provides a pioneer work for future safe RL studies.ADV-PPOL(MR) 525.93±2.

SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles

“…Similarly, the PPO agent that achieves the best route completion (Comp) score presents, however, the highest RR and SS scores, which means that it may run red lights and stop signs most frequently. This observation suggests the inherent contradiction between some safety metrics and functionality metrics, which is also unveiled in some previous studies [46,38,39].…”

Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability

“…Beyond robotics continuous control tasks and simulated games, robust RL is also tested in mobile robot tasks and autonomous driving scenarios. Liu et al [108] propose a safe and robust benchmark containing mobile robot tasks based on Bullet safety gym [60] environments. Jaafra et al [77] propose to test in CARLA simulator [41] with different conditions, including the traffic density, such as the number of dynamic objects, and visual effects such as weather and lightening conditions.…”

“…To better describe the unique properties of a safe RL problem, we provide the feasibility, optimality, and temptation definitions following the previous work [108]. Their figure illustrations for one CMDP are presented in Fig.…”

“…We could observe that improving any aspects of trustworthiness might potentially induce a drop in the best possible task performance. For instance, improving the safety of an RL agent may lead the agent to be conservative in exploring high-rewarding regions, and thus has relatively lower task performance than the unsafe one [108]; increasing the robustness against adversarial perturbations may over-smooth the policy, and thus decrease the task performance; training a generalizable policy on multiple tasks may decrease the performance on a single task due to the limitation of model capacity. As a result, we can see that improving the trustworthiness may be at the cost of sacrificing the optimal task performance, which is also a reflection of the no-free-lunch theorem.…”

“…It has rich meanings that go beyond its literal sense, and motivates a comprehensive framework that includes multiple principles, requirements, and criteria [3]. Recently, there has been exciting progress in the area of trustworthy RL [2,5,48,107,108,121,129,137,140,145,148,165,171,201], which greatly help to advance our understanding of intrinsic vulnerabilities in RL and potential solutions in particular aspects of trustworthy RL. It is clear that the next leap toward trustworthy RL will require a holistic and fundamental understanding of the challenges of these problems, the weakness, and advantages of existing trustworthy RL approaches, and a paradigm shift of trustworthy RL based on existing work.…”