Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees

Abstract

Safety is a critical component of autonomous systems and remains a challenge for learning-based policies to be utilized in the real world. In particular, policies learned using reinforcement learning often fail to generalize to novel environments due to unsafe behavior. In this paper, we propose Sim-to-Lab-to-Real to safely close the reality gap. To improve safety, we apply a dual policy setup where a performance policy is trained using the cumulative task reward and a backup (safety) policy is trained by solving the reach-avoid Bellman Equation based on Hamilton-Jacobi reachability analysis. In Sim-to-Lab transfer, we apply a supervisory control scheme to shield unsafe actions during exploration; in Lab-to-Real transfer, we leverage the Probably Approximately Correct (PAC)-Bayes framework to provide lower bounds on the expected performance and safety of policies in unseen environments. We empirically study the proposed framework for ego-vision navigation in two types of indoor environments including a photo-realistic one. We also demonstrate strong generalization performance through hardware experiments in real indoor spaces with a quadrupedal robot. See https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary material.

Failure Prediction with Statistical Guarantees for Vision-Based Robot Control

“…Our method can thus be used in a 'plug-and-play' manner to improve the safety of an existing policy. One recent work [47] also improves safety of the policies trained using PAC-Bayes theory and detects failure by learning a safety value function using Hamilton-Jacobi reachability-based reinforcement learning, but it does not explicitly provide guarantees on failure prediction.…”

Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles

“…Since autonomous vehicles require interactions with other road users and are safetycritical by nature, there has been a lot of effort, from regulators, industry, and researchers alike, in ensuring safe AV operations [7,8,9,10,11,12,13,14,15,16,17] (see [18] for a review). A common approach is to compute "inevitable" collision sets (ICS), e.g., via Hamilton-Jacobi reachability computation [19], with selected assumptions on other agents' behaviors, and perform shielding where the AV will flag a situation as unsafe whenever it is close to entering the ICS and execute an appropriate evasive action (e.g., [1,20,21]). A primary challenge is selecting reasonable behavior assumptions for ICS computation to balance tractability, interpretability, and compatibility with real-world driving interactions [22,23].…”