Safe Learning for Control using Control Lyapunov Functions and Control Barrier Functions: A Review

Anand, Akhil S; Seel, Katrine; Gjærum, Vilde B.; Håkansson, Anne; Robinson, Haakon; Saad, Aya

doi:10.1016/j.procs.2021.09.173

Cited by 12 publications

(6 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approach proposed in [44] by designing a quadratic Lyapunov candidate function could be coupled with GP models to provide probabilistic stability guarantees similar to safety guarantees in [45]. GP models allow for providing such guarantees on safety and stability by using additional optimization constraints [46]. But this needs further research in the case of VILC for providing closed-loop safety and stability guarantees during learning and for the final policy.…”

Section: Discussionmentioning

confidence: 99%

Data-Efficient Reinforcement Learning for Variable Impedance Control

Anand,

Kaushik,

Gravdahl

et al. 2024

IEEE Access

View full text Add to dashboard Cite

One of the most crucial steps toward achieving human-like manipulation skills in robots is to incorporate compliance into the robot controller. Compliance not only makes the robot's behaviour safe but also makes it more energy efficient. In this direction, the variable impedance control (VIC) approach provides a framework for a robot to adapt its compliance during execution by employing an adaptive impedance law. Nevertheless, autonomously adapting the compliance profile as demanded by the task remains a challenging problem to be solved in practice. In this work, we introduce a reinforcement learning (RL)-based approach called DEVILC (Data-Efficient Variable Impedance Learning Controller) to learn the variable impedance controller through real-world interaction of the robot. More concretely, we use a modelbased RL approach in which, after every interaction, the robot iteratively learns a probabilistic model of its dynamics using the Gaussian process regression model. The model is then used to optimize a neural-network policy that modulates the robot's impedance such that the long-term reward for the task is maximized. Thanks to the model-based RL framework, DEVILC allows a robot to learn the VIC policy with only a few interactions, making it practical for real-world applications. In simulations and experiments, we evaluate DEVILC on a Franka Emika Panda robotic manipulator for different manipulation tasks in the Cartesian space. The results show that DEVILC is a promising direction toward autonomously learning compliant manipulation skills directly in the real world through interactions. A video of the experiments is available in the link: https://youtu.be/_uyr0Vye5no INDEX TERMS Model-based reinforcement learning, Variable impedance learning control, Gaussian processes, Covariance matrix adaptation.

show abstract

Section: Discussionmentioning

confidence: 99%

Data-Efficient Reinforcement Learning for Variable Impedance Control

Anand,

Kaushik,

Gravdahl

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…With the boom of NN control, stability and safety issues of NN-controlled systems are drawing more and more attention in the control and AI community (Anand et al 2021;Dawson, Gao, and Fan 2022;Everett 2021). Compared to the research on safe NN controllers, formal verification for pure BNN or BNN-controlled systems is much newer and there are a few works on this topic: (Wicker et al 2020) studied the probabilistic safety for BNNs, by computing the probability of weights w.r.t.…”

Section: Related Workmentioning

confidence: 99%

Safety Verification of Nonlinear Systems with Bayesian Neural Network Controllers

Zeng

Yang

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

Bayesian neural networks (BNNs) retain NN structures with a probability distribution placed over their weights. With the introduced uncertainties and redundancies, BNNs are proper choices of robust controllers for safety-critical control systems. This paper considers the problem of verifying the safety of nonlinear closed-loop systems with BNN controllers over unbounded-time horizon. In essence, we compute a safe weight set such that as long as the BNN controller is always applied with weights sampled from the safe weight set, the controlled system is guaranteed to be safe. We propose a novel two-phase method for the safe weight set computation. First, we construct a reference safe control set that constraints the control inputs, through polynomial approximation to the BNN controller followed by polynomial-optimization-based barrier certificate generation. Then, the computation of safe weight set is reduced to a range inclusion problem of the BNN on the system domain w.r.t. the safe control set, which can be solved incrementally and the set of safe weights can be extracted. Compared with the existing method based on invariant learning and mixed-integer linear programming, we could compute safe weight sets with larger radii on a series of linear benchmarks. Moreover, experiments on a series of widely used nonlinear control tasks show that our method can synthesize large safe weight sets with probability measure as high as 95% even for a large-scale system of dimension 7.

show abstract

“…Starting at a state (x 0 ), the Lyapunov function of a stable system will asymptotically decrease towards a stable state due to the negative Lyapunov derivative ( V (x t , u t ) < 0). To learn a stable controller, control Lyapunov function (CLF) has been proposed; especially for learning-based controllers [32]. To guarantee asymptotic convergence with rate λ (λ > 0), ( 26) should be satisfied.…”

Section: Control Lyapunov Functionmentioning

confidence: 99%

“…Periodically, a batch of m experiences is used to optimize the two networks using gradient descent with rates l r Q and l r π . First, the critic is updated by minimizing the quadratic loss (L Q ) between the expected Q-value of each of the n output Q-values and a target value (y t ) as in (32). The target y t is approximated from the Bellman form in (24).…”

Section: F Sac Training Proceduresmentioning

confidence: 99%

Robust Speed Control of Ultrasonic Motors Based on Deep Reinforcement Learning of a Lyapunov Function

2022

View full text Add to dashboard Cite

Speed control of ultrasonic motors (USM) needs to be precise, fast, and robust; however, this becomes a challenging task due to the nonlinear behavior of these motors including nonlinear response, pull-out phenomenon, and speed hysteresis. However, linear controllers would be suboptimal and unstable, and nonlinear controllers would require expert knowledge, expensive online calculations, or costly model estimation. In this paper, we propose a model-free nonlinear offline controller that can significantly mitigate these challenges. Using deep reinforcement learning (DRL) algorithms, a neural network speed controller was optimized. A soft actor-critic (SAC) DRL algorithm was chosen due to its sample efficiency, fast convergence, and stable learning. To ensure controller stability, a custom control Lyapunov reward function was proposed. The steady-state USM behavior was mathematically modeled for easing controller design under simulation. The SAC agent was designed and trained first in simulation and then further trained experimentally. The experimental results support that the trained controller can successfully expand speed operation range ([0,300] rpm), plan optimal control trajectories, and stabilize performance under varying load torque and temperature drift.

show abstract

Safe Learning for Control using Control Lyapunov Functions and Control Barrier Functions: A Review

Cited by 12 publications

References 31 publications

Data-Efficient Reinforcement Learning for Variable Impedance Control

Data-Efficient Reinforcement Learning for Variable Impedance Control

Safety Verification of Nonlinear Systems with Bayesian Neural Network Controllers

Robust Speed Control of Ultrasonic Motors Based on Deep Reinforcement Learning of a Lyapunov Function

Contact Info

Product

Resources

About