Off-policy integral reinforcement learning optimal tracking control for continuous-time chaotic systems

Wei, Qinglai; Song, Ruizhuo; Sun, Qiuye; Xiao, Wendong

doi:10.1088/1674-1056/24/9/090504

Cited by 9 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, Wei et al used an algorithm based on reinforcement learning to realize the optimal tracking control for unknown chaotic systems. [22,23] The advantage of the method is its simplicity and easy implementation.…”

Section: Introductionmentioning

confidence: 99%

Control of chaos in Frenkel–Kontorova model using reinforcement learning*

Lei

Han

2021

Chinese Phys. B

View full text Add to dashboard Cite

It is shown that we can control spatiotemporal chaos in the Frenkel–Kontorova (FK) model by a model-free control method based on reinforcement learning. The method uses Q-learning to find optimal control strategies based on the reward feedback from the environment that maximizes its performance. The optimal control strategies are recorded in a Q-table and then employed to implement controllers. The advantage of the method is that it does not require an explicit knowledge of the system, target states, and unstable periodic orbits. All that we need is the parameters that we are trying to control and an unknown simulation model that represents the interactive environment. To control the FK model, we employ the perturbation policy on two different kinds of parameters, i.e., the pendulum lengths and the phase angles. We show that both of the two perturbation techniques, i.e., changing the lengths and changing their phase angles, can suppress chaos in the system and make it create the periodic patterns. The form of patterns depends on the initial values of the angular displacements and velocities. In particular, we show that the pinning control strategy, which only changes a small number of lengths or phase angles, can be put into effect.

show abstract

Section: Introductionmentioning

confidence: 99%

Control of chaos in Frenkel–Kontorova model using reinforcement learning*

Lei

Han

2021

Chinese Phys. B

View full text Add to dashboard Cite

show abstract

“…The iterative value functions and control laws are obtained iteratively, and the iterative control laws must stabilize the system. [7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24] An initial stabilizing control law is required, however, it is often difficult to obtain. While in most applications, fewer iterations are required, and computationally demanding is more than that of the VI iteration algorithm.…”

Section: Introductionmentioning

confidence: 99%

A novel stable value iteration-based approximate dynamic programming algorithm for discrete-time nonlinear systems

Qu¹,

Lin

2018

Chinese Phys. B

View full text Add to dashboard Cite

The convergence and stability of a value-iteration-based adaptive dynamic programming (ADP) algorithm are considered for discrete-time nonlinear systems accompanied by a discounted quadric performance index. More importantly than sufficing to achieve a good approximate structure, the iterative feedback control law must guarantee the closed-loop stability. Specifically, it is firstly proved that the iterative value function sequence will precisely converge to the optimum. Secondly, the necessary and sufficient condition of the optimal value function serving as a Lyapunov function is investigated. We prove that for the case of infinite horizon, there exists a finite horizon length of which the iterative feedback control law will provide stability, and this increases the practicability of the proposed value iteration algorithm. Neural networks (NNs) are employed to approximate the value functions and the optimal feedback control laws, and the approach allows the implementation of the algorithm without knowing the internal dynamics of the system. Finally, a simulation example is employed to demonstrate the effectiveness of the developed optimal control method.

show abstract

“…As an online learning method, IRL makes use of the integral terms of the system cost during different time intervals, which can be addressed as a kind of reinforcement information, to estimate the cost functions, and thus, parts of the system dynamics can be unknown . Based on the IRL method, several works() devoted to solving the continuous‐time optimal control problems. However, these works are all based on policy iteration schemes, in which cases the determination of the initial admissible policies is still difficult.…”

Section: Introductionmentioning

confidence: 99%

Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

Zhang

et al. 2017

Optim Control Appl Methods

View full text Add to dashboard Cite

Summary In this paper, a modified value iteration–based approximate dynamic programming method is proposed for a class of affine nonlinear continuous‐time systems, whose dynamics are partially unknown. The value iteration algorithm is established in an online fashion, and the convergence proof is given. To attenuate the effect caused by the unascertained characteristics of the system dynamics, the integral reinforcement learning scheme is also used. In the proposed approximate dynamic programming method, it is emphasized that the single‐network structure is utilized to estimate the value functions and the control policies. That is, the iteration process is implemented on the actor/critic structure, in which case only the critic NN is required to be identified. Then, the least‐squares scheme is derived for the NN weights updating. Finally, a linear system and a nonlinear system are tested to evaluate the performance of the proposed online value iteration algorithm. Both of the examples show the feasibility and effectiveness of the proposed algorithms.

show abstract

Off-policy integral reinforcement learning optimal tracking control for continuous-time chaotic systems

Cited by 9 publications

References 34 publications

Control of chaos in Frenkel–Kontorova model using reinforcement learning*

Control of chaos in Frenkel–Kontorova model using reinforcement learning*

A novel stable value iteration-based approximate dynamic programming algorithm for discrete-time nonlinear systems

Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

Contact Info

Product

Resources

About