Lyapunov-stable neural-network control

Abstract

Deep learning has had a far reaching impact in robotics. Specifically, deep reinforcement learning algorithms have been highly effective in synthesizing neural-network controllers for a wide range of tasks. However, despite this empirical success, these controllers still lack theoretical guarantees on their performance, such as Lyapunov stability (i.e., all trajectories of the closed-loop system are guaranteed to converge to a goal state under the control policy). This is in stark contrast to traditional model-based controller design, where principled approaches (like LQR) can synthesize stable controllers with provable guarantees. To address this gap, we propose a generic method to synthesize a Lyapunov-stable neural-network controller, together with a neural-network Lyapunov function to simultaneously certify its stability. Our approach formulates the Lyapunov condition verification as a mixed-integer linear program (MIP). Our MIP verifier either certifies the Lyapunov condition, or generates counter examples that can help improve the candidate controller and the Lyapunov function. We also present an optimization program to compute an inner approximation of the region of attraction for the closed-loop system. We apply our approach to robots including an inverted pendulum, a 2D and a 3D quadrotor, and showcase that our neural-network controller outperforms a baseline LQR controller. The code is open sourced at https://github.com/StanfordASL/neural-network-lyapunov.

Robot Learning From Randomized Simulations: A Review

“…We can for example restrict the policy to obey stable system dynamics derived from first principles ( Greydanus et al, 2019 ; Lutter et al, 2019 ). Another approach is to design the model class such that the closed-loop system is passive for all parameterizations of the learned policy, thus guaranteeing stability in the sense of Lyapunov as well as bounded output energy given bounded input energy ( Brogliato et al, 2007 ; Yang et al, 2013 ; Dai et al, 2021 ). All these methods would require significant exploration in the environment, making it even more challenging to learn successful controllers in the real-world directly.…”

Bridging Model-based Safety and Model-free Reinforcement Learning through System Identification of Low Dimensional Linear Models

“…c) Learned Certifications for Model-based Controllers: Remaining methods for connecting safety and learning are to provide certification on learned stability and constraint set using existing model-based controllers. Stability criterion can be achieved by Lyapunov analysis on region of attraction (RoA) [8, 20,52] or by Lipschitz-based safety [32] during training. Safety with constraint set certifications can be learned using control synthesis such as feedback linearization controllers [68], CBFs [46,53,54,55,61,64,71], and Hamilton-Jacobi (HJ) reachability analysis [5,23,31].…”

Generating Stable and Collision-Free Policies through Lyapunov Function Learning

“…The quadratic Lyapunov candidate function (orange) violates the constraint in (2), since avoiding the obstacles implies moving away from the goal, while a valid Lyapunov function (blue) is monotonically decreasing along the demonstration path (red). Instead of explicitly specifying the Lyapunov candidate function, prior work [12] learns a function that satisfies the Lyapunov conditions in (1)- (3). In this work, we will adapt this structure to learn the Lyapunov function.…”

“…We follow the formulation in [12] to structure a Lyapunov candidate function as follows: The policy can be taken to be related to the negative gradient of the learned Lyapunov function…”

“…However, [10], [11] relies on a chosen Lyapunov candidate function to be valid for the demonstrations. Another prior work constrains the Lyapunov stability conditions on neural networks and learns the Lyapunov function with the policy from training data using mixed-integer programming [12]. On the Reinforcement Learning (RL) front, prior work introduces the stability constraint during training by widening the region of attraction of a Lyapunov function [13].…”