Reachability Analysis of Neural Feedback Loops

Abstract

Neural Networks (NNs) can provide major empirical performance improvements for closedloop systems, but they also introduce challenges in formally analyzing those systems' safety properties. In particular, this work focuses on estimating the forward reachable set of neural feedback loops (closed-loop systems with NN controllers). Recent work provides bounds on these reachable sets, but the computationally tractable approaches yield overly conservative bounds (thus cannot be used to verify useful properties), and the methods that yield tighter bounds are too intensive for online computation. This work bridges the gap by formulating a convex optimization problem for the reachability analysis of closed-loop systems with NN controllers. While the solutions are less tight than previous (semidefinite program-based) methods, they are substantially faster to compute, and some of those computational time savings can be used to refine the bounds through new input set partitioning techniques, which is shown to dramatically reduce the tightness gap. The new framework is developed for systems with uncertainty (e.g., measurement and process noise) and nonlinearities (e.g., polynomial dynamics), and thus is shown to be applicable to real-world systems. To inform the design of an initial state set when only the target state set is known/specified, a novel algorithm for backward reachability analysis is also provided, which computes the set of states that are guaranteed to lead to the target set. The numerical experiments show that our approach (based on linear relaxations and partitioning) gives a 5× reduction in conservatism in 150× less computation time compared to the state-of-the-art. Furthermore, experiments on quadrotor, 270-state, and polynomial systems demonstrate the method's ability to handle uncertainty sources, high dimensionality, and nonlinear dynamics, respectively.

Backward Reachability Analysis for Neural Feedback Loops

“…Forward reachability analysis is the focus of many recent works [14]- [21], but while more traditional approaches to reachability analysis, e.g., Hamilton-Jacobi methods, can easily switch from forward to backward with a change of variables [23], [24], backward reachability for NFLs is less straightforward. One challenge with propagating sets backward through a NN is that many activation functions have finite range, meaning that there is not a one-to-one mapping of inputs to outputs (e.g., ReLU(x) = 0 corresponds to all values of x ≤ 0), which can cause large amounts of conservativeness in the BP set estimate as there is an infinite set of possible inputs.…”

“…Alternatively, while [22] analyzes NFLs, they use a quantized state approach [13] that requires an alteration of the original NN through a preprocessing step that can affect its overall behavior. Finally, previous work by the authors [21] derives a closed-form equation that can be used to find under-approximations of the BP set, but this is most useful for goal checking when it is desirable to guarantee that all states in the BP estimate will reach the target set. This work builds off of [21], adapting some of the steps used to find BP under-approximations to instead find BP over-approximations, which are good for obstacle avoidance because they contain all the states that reach the target set.…”

“…Numerous tools have recently been developed to analyze both NNs in isolation [5]- [13] and neural feedback loops (NFLs), e.g., closed-loop systems with NN control policies, [14]- [22]. While many of these tools focus on forward reachability [14]- [21], which certifies safety by estimating where the NN will drive the system, this work focuses on backward reachability [22], as shown in Fig. 1a.…”

“…While forward and backward reachability differ only by a change of variables for systems without NNs [23]- [25], both the nonlinearities and dimensions of the matrices associated with NN controllers lead to fundamental challenges that make propagating sets backward through an NFL complicated. Despite promising prior work [12], [21], [22], there are no existing techniques that efficiently find BP set estimates over multiple timesteps for the general class of linear NFLs considered in this work. Our work addresses this issue by using a series of NN relaxations to constrain a set of linear programs (LPs) that can be used to find BP set approximations that are guaranteed to contain the true BP set.…”

“…• BReach-LP: an LP-based technique to efficiently find multi-step BP over-approximations for NFLs that can be used to guarantee that the system will avoid collisions, • ReBReach-LP: an algorithm to refine multiple one-step BP over-approximations, reducing conservativeness in the BP estimate by up to 88%, • Numerical experiments that exhibit our BP estimation techniques with a double integrator model and a demonstration certifying safety for a linearized ground robot model whereas the forward reachability tools proposed in [20], [21] fail.…”