Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning

Abstract

Reach-avoid optimal control problems, in which the system must reach certain goal conditions while staying clear of unacceptable failure modes, are central to safety and liveness assurance for autonomous robotic systems, but their exact solutions are intractable for complex dynamics and environments. Recent successes in the use of reinforcement learning methods to approximately solve optimal control problems with performance objectives make their application to certification problems attractive; however, the Lagrange-type objective (cumulative costs or rewards over time) used in reinforcement learning is not suitable to encode temporal logic requirements. Recent work has shown promise in extending the reinforcement learning machinery to safety-type problems, whose objective is not a sum, but a minimum (or maximum) over time. In this work, we generalize the reinforcement learning formulation to handle all optimal control problems in the reach-avoid category. We derive a time-discounted reach-avoid Bellman backup with contraction mapping properties and prove that the resulting reach-avoid Q-learning algorithm converges under analogous conditions to the traditional Lagrange-type problem, yielding an arbitrarily tight conservative approximation to the reach-avoid set. We further demonstrate the use of this formulation with deep reinforcement learning methods, retaining zero-violation guarantees by treating the approximate solutions as untrusted oracles in a model-predictive supervisory control framework. We evaluate our proposed framework on a range of nonlinear systems, validating the results against analytic and numerical solutions, and through Monte Carlo simulation in previously intractable problems. Our results open the door to a range of learning-based methods for safe-and-live autonomous behavior, with applications across robotics and automation.

Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees

“…Chow et al [8] use Lagrangian methods to transform the constrained optimization into an unconstrained one over the primal variable (policy) and the dual variable (penalty coefficient). A recent line of works building on reachability analysis argues that optimizing the sum of rewards and penalties is not an accurate encoding of safety [10,11]. Instead, they propose to use RL to find a value function, which is an approximate solution of a Hamilton-Jacobi partial differential equation [22,23].…”

“…Instead, they propose to use RL to find a value function, which is an approximate solution of a Hamilton-Jacobi partial differential equation [22,23]. Hsu et al [11] shows that reachability-based RL has fewer safety violations during deployment compared to standard RL.…”

“…Failures are usually catastrophic in safety-critical settings; thus worst-case safety, instead of an average safety over the trajectory, should be considered. For training the backup policy, we incorporate tools from reachability-based reinforcement learning [10,11] and optimize the discounted safety Bellman equation (DSBE) as below,…”

“…The backup policy is pre-trained in the Sim stage and ready to ensure safe exploration once Lab training starts. Based on safe RL training using Hamilton-Jacobi (HJ) reachability-analysis developed in [10,11], our backup agent can learn from near failure with dense signals; even when the backup policy updates itself in safety-critical conditions, the training does not rely on safety violations unlike previous work that uses binary safety indicators [12,13]. As we show in Sec.…”

Refining Control Barrier Functions through Hamilton-Jacobi Reachability

“…However, directly learning HJ value functions by leveraging neural networks for supervised approximate DP does not provide hard guarantees [18]. Another approach, which leverages the similarity of HJ reachability and reinforcement learning does not necessarily encode safety [19].…”