Off-Policy: Model-Free Optimal Synchronization Control for Complex Dynamical Networks

Wang, Jianfeng; Wang, Yan; Ji, Zhicheng

doi:10.1007/s11063-022-10748-2

Cited by 5 publications

(3 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Usually, the key issue of the optimal control for nonlinear systems is the solution of the Hamilton-Jacobi-Bellman (HJB) equation, which is generally hard to solve analytically since it is a first-order nonlinear partial differential equation. To address the issue, Werbos introduced adaptive dynamic programming (ADP) theory which provided a way to compute the approximate value of the HJB equation online and has attracted a great deal of attention from researchers [14][15][16]. The ADP-based methods typically employ neural networks to build actor-critic architecture, which involves a critic network to approximate performance index function and an actor network to approximate control policy [17,18].…”

Section: Introductionmentioning

confidence: 99%

Adaptive dynamic programming-based optimal control for nonlinear state constrained systems with input delay

Wang,

Zhang,

Wang

et al. 2023

Nonlinear Dyn

View full text Add to dashboard Cite

This paper investigates the problem of adaptive optimal tracking control for strict-feedback nonlinear full-state constrained systems with input delay. A control method combined with the backstepping design technique and adaptive dynamic programming (ADP) theory is developed. An appropriate Barrier Lyapunov Function is implemented into the backstepping design approach to solve the state constraints. To avoid the influence of input delay, we design an intermediate variable using Pade approximation. With regard to uncertainty, neural networks are utilized to approximate the unknown functions. Then, an adaptive backstepping feedforward controller is designed. Based on the feedforward controller, the tracking task for strict-feedback systems can be converted into an equivalent regulation issue for the nonlinear systems in affine form. Secondly, a critic network is constructed under the framework of ADP to approximate the solution of Hamilton-Jacobi-Bellman equation. Following that, online learning is applied to obtain the optimal feedback control. Therefore, the whole controller of the nonlinear system is made up of the feedforward and the feedback part. In addition, the closed-loop system is proved to be stable and its states remain within the bounded intervals. Finally, an example is given to illustrate the effectiveness of our control scheme.

show abstract

Section: Introductionmentioning

confidence: 99%

Adaptive dynamic programming-based optimal control for nonlinear state constrained systems with input delay

Wang,

Zhang,

Wang

et al. 2023

Nonlinear Dyn

View full text Add to dashboard Cite

show abstract

“…Off-policy learning, where an agent learns its current target policy while following a different behavior policy, provides an agent with opportunities to accumulate a wealth of knowledge by learning about the effects of different behavioral policies and underpins many practical implementations of reinforcement learning (RL, [3]) [4,5,6]. In many cases, we would prefer off-policy learning to some extent to learn about the greedy policy while exploring [7,8], to enable data to be generated from one behavior policy to evaluate multiple target policies simultaneously [9,10], to improve data efficiency via experience replay [11], or to correct data discrepancies introduced by the distributed computation [12].…”

Section: Introductionmentioning

confidence: 99%

Generalized Gradient Emphasis Learning for Off-Policy Evaluation and Control with Function Approximation

Cao¹,

Liu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Emphatic temporal-difference (TD) learning (Sutton et al. 2016) is a pioneering off-policy reinforcement learning method involving the use of the followon trace. The recently proposed Gradient Emphasis Learning (GEM, Zhang et al. 2020) algorithm is used to fix the problems of unbounded variance and large emphasis approximation error introduced by the followon trace from the perspective of stochastic approximation. In this paper, we rethink GEM and introduce a novel generalized GEM(β) algorithm to learn the true emphasis. The key to the construction of the generalized GEM(β) algorithm is introducing a tunable hyper-parameter β that is not necessarily the same as the discount factor γ to the GEM operator. We then apply the emphasis estimated by the proposed GEM(β) algorithm to the value estimation gradient and the policy gradient, respectively, yielding the corresponding emphatic TD variant for off-policy evaluation and actor-critic algorithm for off-policy control. Finally, we demonstrate empirically the advantage of the proposed algorithms across a range of problems, for both off-policy evaluation and off-policy control, and for both linear and nonlinear function approximation.

show abstract

Section: Introductionmentioning

confidence: 99%

Adaptive Dynamic Programming-Based Optimal Control for Nonlinear State Constrained Systems with Input Delay

Wang

Zhang

Wang

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper investigates the problem of adaptive optimal tracking control for strict-feedback nonlinear full-state constrained systems with input delay. A control method combined with the backstepping design technique and adaptive dynamic programming (ADP) theory is developed. An appropriate Barrier Lyapunov Function is implemented into the backstepping design approach to solve the state constraints. To avoid the influence of input delay, we design an intermediate variable using Pade approximation. With regard to uncertainty, neural networks are utilized to approximate the unknown functions. Then, an adaptive backstepping feedforward controller is designed. Based on the feedforward controller, the tracking task for strict-feedback systems can be converted into an equivalent regulation issue for the nonlinear systems in affine form. Secondly, a critic network is constructed under the framework of ADP to approximate the solution of Hamilton–Jacobi–Bellman equation. Following that, online learning is applied to obtain the optimal feedback control. Therefore, the whole controller of the nonlinear system is made up of the feedforward and the feedback part. In addition, the closed-loop system is proved to be stable and its states remain within the bounded intervals. Finally, an example is given to illustrate the effectiveness of our control scheme.

show abstract

Off-Policy: Model-Free Optimal Synchronization Control for Complex Dynamical Networks

Cited by 5 publications

References 38 publications

Adaptive dynamic programming-based optimal control for nonlinear state constrained systems with input delay

Adaptive dynamic programming-based optimal control for nonlinear state constrained systems with input delay

Generalized Gradient Emphasis Learning for Off-Policy Evaluation and Control with Function Approximation

Adaptive Dynamic Programming-Based Optimal Control for Nonlinear State Constrained Systems with Input Delay

Contact Info

Product

Resources

About