Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

Abstract

Summary In this paper, a modified value iteration–based approximate dynamic programming method is proposed for a class of affine nonlinear continuous‐time systems, whose dynamics are partially unknown. The value iteration algorithm is established in an online fashion, and the convergence proof is given. To attenuate the effect caused by the unascertained characteristics of the system dynamics, the integral reinforcement learning scheme is also used. In the proposed approximate dynamic programming method, it is emphasized that the single‐network structure is utilized to estimate the value functions and the control policies. That is, the iteration process is implemented on the actor/critic structure, in which case only the critic NN is required to be identified. Then, the least‐squares scheme is derived for the NN weights updating. Finally, a linear system and a nonlinear system are tested to evaluate the performance of the proposed online value iteration algorithm. Both of the examples show the feasibility and effectiveness of the proposed algorithms.

A novel optimal control design for unknown nonlinear systems based on adaptive dynamic programming and nonlinear model predictive control

“…Theorem 1. For the model NN (10), if the activation function is the sigmoid function and the hidden-to-output weight is updated by the adaptive law (13), then the approximate error x and wmo is uniformly ultimately bounded.…”

“…The optimal control calculated from our algorithm FIGURE 7 The optimal control from the paper [13] FIGURE 8 The cost function produced in our algorithm (upper) and the optimal cost function (lower) [13]. Finally, the initial admissible control law is obtained from NMPC in proposed algorithm and the initial weights of neural networks can be designed reasonably.…”

“…Adaptive dynamic programming (ADP) is an important part of optimal control theory, which can avoid the "curse of dimensionality" [5,6]. Until to now, the ADP algorithms have many research results in discrete systems [7][8][9][10][11] and continuous systems [12][13][14][15][16][17][18]. Zhang et al [18] introduced a precompensator to construct an augmented system and the Hamilton-Jacobi-Bellman (HJB) equation was solved by the least squared technique, neural network (NN) approximator and policy iteration algorithm.…”

“…Yang et al [16] proposed an output‐based tracking control scheme for a class of continuous‐time nonlinear systems via the adaptive dynamic programming (ADP) technique. For partially unknown continuous‐time nonlinear systems, a online adaptive critic algorithm using least squares support vector machine was presented in Sun et al [14] and a value iteration‐based ADP algorithm was proposed in Su et al [13] to solve the optimal control problem. Wei et al [15] designed the continuous‐time time‐varying iteration algorithm to deal with the optimal control problem for the continuous‐time time‐varying nonlinear system.…”

Applications of integral reinforcement learning control in electrical machines and power converter systems

“…Compared with the existing literature on VI based IRL[64] [65][66], the main contributions of this chapter are enumerated here.…”