Dynamic intermittent Q‐learning–based model‐free suboptimal co‐design of ‐stabilization

Abstract

Summary This paper proposes an intermittent model‐free learning algorithm for linear time‐invariant systems, where the control policy and transmission decisions are co‐designed simultaneously while also being subjected to worst‐case disturbances. The control policy is designed by introducing an internal dynamical system to further reduce the transmission rate and provide bandwidth flexibility in cyber‐physical systems. Moreover, a Q‐learning algorithm with two actors and a single critic structure is developed to learn the optimal parameters of a Q‐function. It is shown by using an impulsive system approach that the closed‐loop system has an asymptotically stable equilibrium and that no Zeno behavior occurs. Furthermore, a qualitative performance analysis of the model‐free dynamic intermittent framework is given and shows the degree of suboptimality concerning the optimal continuous updated controller. Finally, a numerical simulation of an unknown system is carried out to highlight the efficacy of the proposed framework.

Data-Driven Nonzero-Sum Game for Discrete-Time Systems Using Off-Policy Reinforcement Learning

“…One of such algorithms, named off-line policy iteration algorithm, can be introduced as in Algorithm 1. By iteratively solving the Lyapunov equations (29) and (30), which are linear for P i 1 , P i 2 , and updating K i , L i by (31) and (32), as shown at the bottom of this page, the pair (P i 1 , P i 2 ) converges to the solutions (P * 1 , P * 2 ) to the CARE (16) and (17) asymptotically [13].…”

Hamiltonian‐driven adaptive dynamic programming for mixedH2/H∞performance using sum‐of‐squares

“…In a CPS, the plant, the actuators, and the sensors are distributed located, where the signal transmission channels are vulnerable to system uncertainty and adversarial environment. Therefore, it is important to investigate the imperfection for modeling due to the existence of communication delay, 9,10 packet dropout, 11,12 external disturbance input, 13 false‐data injection 14,15 . There are mainly two types of techniques for the modeling and analysis of the imperfection in the CPS.…”

H∞ tracking control for linear discrete‐time systems via reinforcement learning

“…However, most LMI‐based methods need to solve complex linear matrix inequalities and inevitably require the system model information, which cannot be obtained accurately in many practical applications. In addition, similar to H ∞ control, H ∞ tracking control problem can be converted into a two‐person zero‐sum game problem according to game theory, where the controller is a minimizing player and the disturbance is a maximizing player . In this process, finding the saddle point of zero‐sum game by solving game algebraic Riccati equation (GARE) is the key to solving the H ∞ tracking control of a linear system …”

“…In addition, similar to H ∞ control, H ∞ tracking control problem can be converted into a two-person zero-sum game problem according to game theory, where the controller is a minimizing player and the disturbance is a maximizing player. 12,13 In this process, finding the saddle point of zero-sum game by solving game algebraic Riccati equation (GARE) is the key to solving the H ∞ tracking control of a linear system. 14 Reinforcement learning (RL), which emphasizes the use of environment-based feedback to modify behavior, is originally used by Werbos to solve optimal regulator problem in control field.…”