Optimal model-free output synchronization of heterogeneous systems using off-policy reinforcement learning

Abstract

This paper considers optimal output synchronization of heterogeneous linear multi-agent systems. Standard approaches to output synchronization of heterogeneous systems require either the solution of the output regulator equations or the incorporation of a p-copy of the leader's dynamics in the controller of each agent. By contrast, in this paper neither one is needed. Moreover, here both the leader's and the follower's dynamics are assumed to be unknown. First, a distributed adaptive observer is designed to estimate the leader's state for each agent. The output synchronization problem is then formulated as an optimal control problem and a novel model-free off-policy reinforcement learning algorithm is developed to solve the optimal output synchronization problem online in real time. It is shown that this optimal distributed approach implicitly solves the output regulation equations without actually doing so. Simulation results are provided to verify the effectiveness of the proposed approach

Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

“…The convergent proofs of the value iteration-based HDP method and DHP method were shown in the works of Al-Tamimi et al 41 and Zhang et al, 42 which are all addressed in the discrete-time systems. As for the continuous-time cases, it is the convergence of (14) and (15) that is desired to be proved in this work. In the work of Vrabie and Lewis, 34 a similar convergence proof was given.…”

“…Lemma 3. Consider system (1), given V i (x(t)) defined as in (15). If the system is controllable, then, ∀i ∈ N, there exists an upper bound Y such that 0 ≤ V i (x(t)) ≤ Y.…”

“…Lemma 3 shows that positive series {V i } acquired from (15) has an upper bound. Similar results for the discrete-time cases were given in other works.…”

“…The optimal control problems have been studied for more than half a century, and quite a number of methods have been presented, such as fuzzy logic theories [1][2][3][4] and approximation dynamic programming (ADP) algorithms. [5][6][7][8][9][10][11][12][13][14][15][16][17] With the rapid development of intelligent computation technologies in the last several decades, intelligent control methods have been applied in the analysis of nonlinear systems (see the works of Zhang et al, 18 Zhao et al, 19 and Huang and Chung 20 ). For nonlinear systems, the optimal control is to find the solutions to the Hamilton-Jacobi-Bellman (HJB) equation.…”

Consensus of Second Order Multi-Agent Systems with Exogenous Disturbance Generated by Unknown Exosystems

“…The consensus problem of multi-agent systems has received increasing attention in recent years due to its broad applications in such areas as cooperative control of unmanned aircrafts and underwater vehicles, flocking of mobile vehicles, communication among wireless sensor networks, rendezvous, formation control, and so on, see [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15]. In the past years, many researches have been firstly concerned with consensus problems of first order multi-agent systems [16][17][18][19][20].…”