Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear Systems

Abstract

This paper is concerned with a new discrete-time policy iteration adaptive dynamic programming (ADP) method for solving the infinite horizon optimal control problem of nonlinear systems. The idea is to use an iterative ADP technique to obtain the iterative control law, which optimizes the iterative performance index function. The main contribution of this paper is to analyze the convergence and stability properties of policy iteration method for discrete-time nonlinear systems for the first time. It shows that the iterative performance index function is nonincreasingly convergent to the optimal solution of the Hamilton-Jacobi-Bellman equation. It is also proven that any of the iterative control laws can stabilize the nonlinear systems. Neural networks are used to approximate the performance index function and compute the optimal control law, respectively, for facilitating the implementation of the iterative ADP algorithm, where the convergence of the weight matrices is analyzed. Finally, the numerical results and analysis are presented to illustrate the performance of the developed method.

Optimal control for discrete‐time systems with actuator saturation

“…17 And when j approaches the infinity, the developed algorithm becomes a policy iteration. 45 Above all, we can conclude that the developed novel ADP algorithm is a general idea that unifies almost all ADP and reinforcement learning methods.…”

“…Moreover, a control law, which not only stabilizes the system (1) but also make the performance index function finite, is said to be admissible. 45 For simplicity, the systems (1) can be represented as…”

“…17 On the other hand, when j approaches the infinity, the developed algorithm can be considered as a policy iteration algorithm. 45 Furthermore, the developed algorithm can accelerate the convergence rate without requiring to solve the HJB equation for i-iteration. First, a nonquadratic function is used to derive the HJB equation for discrete-time nonlinear systems with actuator saturation.…”

Safe Approximate Dynamic Programming via Kernelized Lipschitz Estimation

“…It is not always clear how to initialize the weights of the neural approximators (26). Commonly, small random numbers drawn from a uniform distribution are used [39], but there is no safety guarantee associated with random initialization. We propose initializing the weights as follows.…”

Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

“…Both of the examples show the feasibility and effectiveness of the proposed algorithms.KEYWORDS approximation dynamic programming (ADP), continuous-time systems, integral reinforcement learning (IRL), online learning, value iteration SU ET AL.heuristic dynamic programming (HDP), action-dependent HDP, dual HDP (DHP), action-dependent DHP, globalized DHP, and action-dependent GDHP. 32,33 In addition, from an implementation point of view, the iteration schemes of ADP can be divided into 2 classes: policy iteration algorithms and value iteration algorithms.The implementation process of the policy iteration method should start with a given initial admissible policy (the definition will be given herein). However, by now, how to obtain an admissible policy is still an open issue.…”