Open‐loop Stackelberg learning solution for hierarchical control problems

Abstract

This work presents a novel framework based on adaptive learning techniques to solve the continuous-time open-loop Stackelberg games. The method yields real-time approximations of the game value and convergence of the policies to the open-loop Stackelberg-equilibrium solution, while also guaranteeing asymptotic stability of the equilibrium point of the closed-loop system. It is implemented as a separate actor/critic parametric network approximator structure for every player and involves simultaneous continuous-time adaptation. To introduce and implement the hierarchical structure to the coupled optimization problem, we adjoin to the leader the controller dynamics of the follower. A persistence of excitation condition guarantees convergence of both critics to the actual game values that eventually solve the hierarchical optimization problem. A simulation example shows the efficacy of the proposed approach. KEYWORDS leader-follower coupled Riccati equations, learning-based adaptation, noncooperative games, Stackelberg equilibrium Int J Adapt Control Signal Process. 2019;33:285-299.wileyonlinelibrary.com/journal/acs

Observer‐based adaptive optimal output containment control problem of linear heterogeneous Multiagent systems with relative output measurements

“…Theorem 2. Consider the leader dynamics ( 4)-( 5), and the adaptive observer ( 35)- (37). Let ̃ = − , ̃ = − be leaders' dynamics estimation errors and ̃ = ̂ − * and ̃ 0 ( ) = ̂ − * be th follower state estimation error of the convex combination of the leaders' states and outputs, respectively, where * is −( −1 1 2 ⊗ ̄ ) with = ( 1 , ..., ).…”

“…Theorem 3. Consider the multi-agent system (2)-( 5) and the distributed adaptive observer (35) along with adaptation laws (36) and (37). Under Assumptions 1 -6, Problem 2 and consequently Problem 1 are solved using optimal control policy (68) with ̄ * given by ( 60) and (61).…”

“…Another important issue, which is not considered in the existing results for containment control, is designing online solutions that do not require having access to the complete knowledge of the leaders by all followers (fourth limitation). Reinforcement learning (RL) 33,34 has been successfully used to design adaptive optimal controllers for single-agent systems [35][36][37][38][39][40][41][42][43] and multiagent systems [44][45][46][47][48] online. However, to our knowledge, there is no RL-based solution to the optimal output containment control problem.…”

Exploring the effects of computational costs in extensive games via modeling and simulation

“…In generative adversarial networks, 24 for instance, two neural networks are essentially playing a competitive game in which they continuously optimize themselves to obtain the Nash equilibrium. Mean field games (MFGs) and stochastic games (SGs) 25 are used to guide the deep reinforcement learning approach. 26 One of the classical models for games where players take turns to move is extensive (-form) games.…”

Editorial for the Special Issue on Learning‐based Adaptive Control: Theory and Applications

“…An off‐policy reinforcement learning algorithm on an actor‐critic structure is developed to solve an optimal containment control problem using only relative output measurements between agents, without the knowledge of the leaders' exact dynamics. Vamvoudakis et al propose a novel learning‐based adaptive solution to a two‐player differential game. More specifically, this paper introduces an actor/critic learning solution to a continuous‐time open‐loop Stackelberg game.…”