Online optimal and adaptive integral tracking control for varying discrete‐time systems using reinforcement learning

Abstract

SummaryConventional closed‐form solution to the optimal control problem using optimal control theory is only available under the assumption that there are known system dynamics/models described as differential equations. Without such models, reinforcement learning (RL) as a candidate technique has been successfully applied to iteratively solve the optimal control problem for unknown or varying systems. For the optimal tracking control problem, existing RL techniques in the literature assume either the use of a predetermined feedforward input for the tracking control, restrictive assumptions on the reference model dynamics, or discounted tracking costs. Furthermore, by using discounted tracking costs, zero steady‐state error cannot be guaranteed by the existing RL methods. This article therefore presents an optimal online RL tracking control framework for discrete‐time (DT) systems, which does not impose any restrictive assumptions of the existing methods and equally guarantees zero steady‐state tracking error. This is achieved by augmenting the original system dynamics with the integral of the error between the reference inputs and the tracked outputs for use in the online RL framework. It is further shown that the resulting value function for the DT linear quadratic tracker using the augmented formulation with integral control is also quadratic. This enables the development of Bellman equations, which use only the system measurements to solve the corresponding DT algebraic Riccati equation and obtain the optimal tracking control inputs online. Two RL strategies are thereafter proposed based on both the value function approximation and the Q‐learning along with bounds on excitation for the convergence of the parameter estimates. Simulation case studies show the effectiveness of the proposed approach.

Regulation With Guaranteed Convergence Rate for Continuous-Time Systems With Completely Unknown Dynamics in the Presence of Disturbance

“…Consequently, the solution to the optimization problem (11) is equivalent to the solution to the min-max optimization problem…”

Optimal event‐triggered control for C‐T system with asymmetric constraints based on dual heuristic dynamic programing structure

“…Approximate optimal solutions are obtained under partially unknown nonlinear systems dynamics and perturbations, which is on the basis of successive approximate solutions of HJI equation. 18 An integral reinforcement learning algorithm is proposed 19,20 for online solving the Nash equilibrium point of ZS differential game with policy iterations when the dynamics of the systems are completely unknown.…”

Assured learning‐enabled autonomy: A metacognitive reinforcement learning framework

“…13 That is, current RL practice has mainly been used to achieve pre-specified goals in structured environments by handcrafting a cost or reward function for which its minimization guarantees reaching the goal. [14][15][16][17][18][19] Strong AI, on the other hand, holds the promise of designing agents that can learn to achieve goals across multiple circumstances by generalizing to unforeseen and novel situations. As the designer cannot foresee all the circumstances that the agent might encounter, pre-specifying and handcrafting the reward function cannot guarantee reaching goals in an uncertain and non-stationary environment.…”