Inverse optimal control and construction of control Lyapunov functions

Safety is a critical factor in reinforcement learning (RL) in chemical processes. In our previous work, we had proposed a new stability-guaranteed RL for unconstrained nonlinear control-affine systems. In the approximate policy iteration algorithm, a Lyapunov neural network (LNN) was updated while being restricted to the control Lyapunov function, and a policy was updated using a variation of Sontag's formula. In this study, we additionally consider state and input constraints by introducing a barrier function, and we extend the applicable type to general nonlinear systems. We augment the constraints into the objective function and use the LNN added with a Lyapunov barrier function to approximate the augmented value function. Sontag's formula input with this approximate function brings the states into its lower level set, thereby guaranteeing the constraints satisfaction and stability. We prove the practical asymptotic stability and forward invariance. The effectiveness is validated using four tank system simulations.

Section: Clf and Sontag's Formulamentioning

confidence: 99%

Safe model‐based reinforcement learning for nonlinear optimal control with state and input constraints

Kim

2022

“…The existence of the CLF is equivalent to the asymptotic stabilizability of the system via a smooth controller 65,66,68 because of the converse Lyapunov theorem 69 and Sontag's formula. In addition, the variation of Sontag's formula given in Equation ) is equivalent to the optimal controller for a user‐defined cost function r ( x , u ) = q ( x ) + u ⊤ Ru when the CLF has the same level set shapes as those of the optimal value function V * 29,57,70,71 :

\begin{matrix} lefttrue \\ ψ (x) = \{\begin{matrix} left \\ - \frac{L_{f} V_{c} + \sqrt{L_{f} V_{c}^{2} + q (x) L_{g} V_{c} R^{- 1} L_{g} V_{c}^{⊤}}}{L_{g} V_{c} R^{- 1} L_{g} V_{c}^{⊤}} R^{- 1} L_{g} V_{c}^{⊤} & L_{g} V_{c} \neq 0 \\ 0 & L_{g} V_{c} = 0 \end{matrix} \end{matrix}

In other words, for a CLF that satisfies V c = α c ( V *) with some differentiable class

scriptK

function α c , ψ ( x ) is equivalent to the optimal controller u * because of the HJB equation and

\frac{\partial V_{c}}{\partial x} = \frac{\partial α_{c} ((), V^{*})}{\partial V^{*}} \frac{\partial V^{*}}{\partial x} = λ ((), x) \frac{\partial V^{*}}{\partial x}

with λ ( x ) > 0 for all x ≠ 0 70 . For L g V ≠ 0,

\begin{matrix} lefttrue \\ - \frac{L_{f} V_{c} + \sqrt{L_{f} V_{c}^{2} + q (x) L_{g} V_{c} R^{- 1} L_{g} V_{c}^{⊤}}}{L_{g} V_{c} R^{- 1} L_{g ...}} \end{matrix}

…”

Section: Preliminariesmentioning

confidence: 99%

Model‐based reinforcement learning for nonlinear optimal control with practical asymptotic stability guarantees

Kim

Lee

2020

We propose a new reinforcement learning approach for nonlinear optimal control where the value function is updated as restricted to control Lyapunov function (CLF) and the policy is improved using a variation of Sontag's formula. The practical asymptotic stability of the closed‐loop system is guaranteed during the training and at the end of training without requiring an additional actor network and its update rule. For a single‐layer neural network (NN) with exact basis functions, the approximate function converges to the optimal value function, resulting in the optimal controller. When a deep NN is used, the level set shapes of the trained NN become similar to those of the optimal value function. Because Sontag's formula with CLF is equivalent to the optimal controller when the given CLF has the same level set shapes as the optimal value function, Sontag's formula with the trained NN provides a nearly optimal controller.

“…Remark 3 Note that while there are currently no general methods for constructing control Lyapunov functions for general nonlinear systems, significant progress has been made on the constructing control Lyapunov functions for different classes of systems including input affine nonlinear systems 35 and constraint linear systems. 36 Note also that for broad classes of nonlinear models arising in the context of chemical process control applications, quadratic Lyapunov functions are widely used and provide very good estimates of closed-loop stability regions.…”

Section: Stabilizability and Observability Assumptionsmentioning

confidence: 99%

Lyapunov‐based MPC with robust moving horizon estimation and its triggered implementation

Zhang

Liu

2013

In this work, we consider moving horizon state estimation (MHE)-based model predictive control (MPC) of nonlinear systems. Specifically, we consider the Lyapunov-based MPC (LMPC) developed in (Mhaskar et al., IEEE Trans Autom Control. 2005;50:1670-1680 Syst Control Lett. 2006;55:650-659) and the robust MHE (RMHE) developed in (Liu J, Chem Eng Sci. 2013;93:376-386). First, we focus on the case that the RMHE and the LMPC are evaluated every sampling time. An estimate of the stability region of the output control system is first established; and then sufficient conditions under which the closed-loop system is guaranteed to be stable are derived. Subsequently, we propose a triggered implementation strategy for the RMHE-based LMPC to reduce its computational load. The triggering condition is designed based on measurements of the output and its time derivatives. To ensure the closed-loop stability, the formulations of the RMHE and the LMPC are also modified accordingly to account for the potential open-loop operation. A chemical process is used to illustrate the proposed approaches.