Discrete-Time Optimal Control via Local Policy Iteration Adaptive Dynamic Programming

Abstract

In this paper, a discrete-time optimal control scheme is developed via a novel local policy iteration adaptive dynamic programming algorithm. In the discrete-time local policy iteration algorithm, the iterative value function and iterative control law can be updated in a subset of the state space, where the computational burden is relaxed compared with the traditional policy iteration algorithm. Convergence properties of the local policy iteration algorithm are presented to show that the iterative value function is monotonically nonincreasing and converges to the optimum under some mild conditions. The admissibility of the iterative control law is proven, which shows that the control system can be stabilized under any of the iterative control laws, even if the iterative control law is updated in a subset of the state space. Finally, two simulation examples are given to illustrate the performance of the developed method.

Radial pole path approach for fast response of affine constrained nonlinear systems with matched uncertainties

“…The following design algorithm provides an appropriate selection of the coefficients a r−1 , … , a 0 in (7) in order to guarantee the satisfaction of the input constraints (4).…”

“…To pursue this goal, a lot of research work has been carried out in the context of optimal control. After the seminal work about time-optimal control of continuous-time linear systems by Pontryagin, 1 which leads to a bang-bang control scheme, many contributions toward the development of time-optimal control for other classes of systems, such as linear discrete-time systems 2,3 as well as both discrete-and continuous-time nonlinear systems, [4][5][6][7][8][9] have appeared in the literature. One of the main problems when implementing time-optimal control is that there is no guarantee that it results into a stable system.…”

A novel triggering condition of event‐triggered control based on heuristic dynamic programming for discrete‐time systems

“…It is hard to solve nonanalytical equations like (7) and (8). Thus, an event-triggered HDP algorithm with a novel triggering condition is studied in the next section to solve the problem.…”

“…Adaptive dynamic programming (ADP) refers to a family of practical actor-critic methods for finding optimal solutions in real time, 1 and it is a self-learning method. [2][3][4][5][6][7][8] In 1977, adaptive critic design was first proposed by Werbos,9 which takes the advantages of neural networks (NNs). Then, several names emerged, eg, approximate dynamic programming and asymptotic dynamic programming.…”

An off‐policy iteration algorithm for robust stabilization of constrained‐input uncertain nonlinear systems

“…The early studies in the field of RL and ADP included the works of Werbos 10 and Sutton. 11 After that, various RL and ADP were reported, such as integral RL, 12,13 online RL, [14][15][16] off-policy RL, [17][18][19] local value/policy iterative ADP, [20][21][22] Hamiltonian-driven ADP, 23 robust ADP, 24,25 and goal representation ADP. 26,27 Over the past several years, RL and ADP have been widely used to solve robust nonlinear control problems.…”