Integral reinforcement learning off-policy method for solving nonlinear multi-player nonzero-sum games with saturated actuator

Ren, He; Zhang, Huaguang; Wen, Yinlei; Liu, Chong

doi:10.1016/j.neucom.2019.01.033

Cited by 26 publications

(6 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the system state x is UUB due to the property of the function ∇L i . Remark 3: Different from the method proposed in [37] that each player needs both critic NN and actor NN, the presented method in this paper requires only critic NN for each player to tackle the coupled HJ equations. By abnegating the actor NN, identifier-critic framework is employed, which can effectively simplify the algorithm complexity and save the computing resources.…”

Section: Stability Analysismentioning

confidence: 99%

“…The method in [36] also utilized event-triggered mechanism to solve nonlinear H ∞ control issues with constraints. In [37], an IRL method was used to figure out the optimal control policies for players in NZS games with saturated actuator. Nevertheless, this method employed both actor NN and critic NN, which perplexed the algorithm and aggravated the computing burdens.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Online Dual-Network-Based Adaptive Dynamic Programming for Solving Partially Unknown Multi-Player Non-Zero-Sum Games With Control Constraints

et al. 2020

Self Cite

View full text Add to dashboard Cite

In this paper, a novel online method for multi-player non-zero-sum (NZS) differential games of nonlinear partially unknown continuous time (CT) systems with control constraints is developed based on neural networks (NN). The issue of multi-player NZS games with saturated actuator is elaborately analyzed and the unknown dynamics model is learned by applying identifier NN. Different from using the standard identifier-actor-critic framework of adaptive dynamic programming (ADP), the proposed method uses only identifier networks and critic networks for all the players to solve the coupled Hamilton-Jacobi (HJ) equations for multi-player NZS games, which could effectively simplify the algorithm and save computing resources. Moreover, a tuning law which utilizes the gradient descent method is designed for each critic network. Meanwhile, to remove the requirement for the initial stabilizing control, a novel stability term is designed to ensure the system stability during the training phase of the critic NN. By the means of Lyapunov approach, it is proven that the system states, the critic network weight estimation errors and the obtained control are all uniformly ultimately bounded (UUB). Finally, two numerical examples are simulated to illustrate the validity of the developed method for multi-player NZS games with control constraints. INDEX TERMS Adaptive critic designs, adaptive dynamic programming, control constraints, multi-player, non-zero-sum games.

show abstract

Section: Stability Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Online Dual-Network-Based Adaptive Dynamic Programming for Solving Partially Unknown Multi-Player Non-Zero-Sum Games With Control Constraints

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Considering the constrained control inputs and partially unknown dynamics, 42 designs an online ADP approach to solve the finite‐horizon optimal control for NZS game. Consider unknown system dynamics and saturated actuator, an off‐policy IRL learning mechanism is introduced to obtain the solution of nonlinear NZS game 43 . Reference 44 proposes a data‐driven IRL method to solve the nonlinear optimization problem of NZS games.…”

Section: Introductionmentioning

confidence: 99%

Off‐policy integral reinforcement learning‐based optimal tracking control for a class of nonzero‐sum game systems with unknown dynamics

Zhao

Chen

2022

Optim Control Appl Methods

View full text Add to dashboard Cite

This article studies the optimal tracking control problem of a class of multi‐input nonlinear system with unknown dynamics based on reinforcement learning (RL) and nonzero‐sum game theory. First of all, an augmented system composed of the tracking error dynamics and the command generator dynamics is constructed. Then, a tracking coupled Hamilton–Jacobi (HJ) equations associated with discounted cost function is derived, which gives the Nash equilibrium solution. The existence of Nash equilibrium is proved. To approximate the Nash equilibrium solution of tracking coupled HJ equations, we give two model‐based policy iteration (PI) algorithms, and analyze their equivalence and convergence. Further, to get rid of the prior knowledge of system dynamics, an off‐policy integral reinforcement learning (OP‐IRL) algorithm implemented by neural networks (NNs) is proposed. The weights of critic NNs and actor NNs are updated simultaneously by the gradient descent method. The convergence of the NNs weights and the stability of the closed‐loop error systems are proved. Finally, numerical simulation results are provided to demonstrate the effectiveness of the proposed OP‐IRL method.

show abstract

“…Thence, IRL can also be classified into iterative methods for solving the HJB equations. For example, in Ren et al (2019), IRL technology was employed to settle the NZS game in ODE system when the actuators are constrained.…”

Section: Introductionmentioning

confidence: 99%

Off-policy integral reinforcement learning algorithm in dealing with nonzero sum game for nonlinear distributed parameter systems

Ren

Dai

Zhang

et al. 2020

Transactions of the Institute of Measurement and Control

Self Cite

View full text Add to dashboard Cite

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

show abstract

Integral reinforcement learning off-policy method for solving nonlinear multi-player nonzero-sum games with saturated actuator

Cited by 26 publications

References 24 publications

Online Dual-Network-Based Adaptive Dynamic Programming for Solving Partially Unknown Multi-Player Non-Zero-Sum Games With Control Constraints

Online Dual-Network-Based Adaptive Dynamic Programming for Solving Partially Unknown Multi-Player Non-Zero-Sum Games With Control Constraints

Off‐policy integral reinforcement learning‐based optimal tracking control for a class of nonzero‐sum game systems with unknown dynamics

Off-policy integral reinforcement learning algorithm in dealing with nonzero sum game for nonlinear distributed parameter systems

Contact Info

Product

Resources

About