Model‐based reinforcement learning for nonlinear optimal control with practical asymptotic stability guarantees

Abstract

We propose a new reinforcement learning approach for nonlinear optimal control where the value function is updated as restricted to control Lyapunov function (CLF) and the policy is improved using a variation of Sontag's formula. The practical asymptotic stability of the closed‐loop system is guaranteed during the training and at the end of training without requiring an additional actor network and its update rule. For a single‐layer neural network (NN) with exact basis functions, the approximate function converges to the optimal value function, resulting in the optimal controller. When a deep NN is used, the level set shapes of the trained NN become similar to those of the optimal value function. Because Sontag's formula with CLF is equivalent to the optimal controller when the given CLF has the same level set shapes as the optimal value function, Sontag's formula with the trained NN provides a nearly optimal controller.

Safe model‐based reinforcement learning for nonlinear optimal control with state and input constraints

“…Thus, for an arbitrary small , we can set the compact set by excluding the arbitrary thin boundary layer from the . Then, there exists an such that if satisfies the CLF condition on grid points, then is a CLF function on the domain , which is proven in proposition 1 in Reference 39.Theorem Given a constrained set defined by Equation () with a continuously differentiable function , the system in Equation () is practically asymptotically stable on under the controller for all k and for an arbitrary small . With the largest , is the estimate of ROA.…”

“…The similarity of the level-set shapes between two scalar functions can be represented by calculating the standard deviation of the element-wise division of their gradient vectors. 39 If we precisely know the optimal value function, this measure can be used to demonstrate the similarity degree of the trained CLF and the optimal value function. However, determining the optimal value function is difficult, which is why we use RL to learn the optimal control policy along with the optimal value function.…”

“…The similarity of the level‐set shapes between two scalar functions can be represented by calculating the standard deviation of the element‐wise division of their gradient vectors 39 . If we precisely know the optimal value function, this measure can be used to demonstrate the similarity degree of the trained CLF and the optimal value function.…”

“…In our previous study, 39 we proposed a new stability‐oriented RL for control‐affine nonlinear systems. It guarantees practical asymptotic stability during training and at the end of training.…”

Using process data to generate an optimal control policy via apprenticeship and reinforcement learning

“…Instead, RL learns from experience of the process, allowing for π(Á) to be recalibrated as the process evolves through time via process data. 5 Furthermore, RL has shown significant industrial potential as demonstrated in a number of research works, which have explored application to the calibration of PID controllers; 6 set point tracking; 7 dynamic optimization of nonlinear, stochastic systems; 5,8,9 de novo drug 10 and protein design; 11 and in augmentation of the performance of various model predictive control (MPC) approaches. 12,13 Indeed, the potential use of RL draws discussion of its relation to MPC in the development of APC schemes.…”