Continuous-Time Q-Learning for Infinite-Horizon Discounted Cost Linear Quadratic Regulator Problems

Muthukumar, P.; Modares, Hamidreza; Lewis, Frank L.; Aurangzeb, Muhammad

doi:10.1109/tcyb.2014.2322116

Cited by 83 publications

(37 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, Algorithm 2 is completely model-free (data-driven) and robust to the inaccuracy in system modeling. Note that once the rank condition is met, the data of the state and behavior polices can be used repeatedly, which is more computationally efficient than the existing Q-learning methods [41,44,48,49], for which new experience data must be regenerated in each new iteration.…”

Section: Remark 4 Though the Initially Admissible Feedback Gainsmentioning

confidence: 99%

“…However, there are very few studies on continuous-time systems, because it is difficult to construct the Q-functions for continuous-time systems. In [44], a Qlearning method was developed for infinite-horizon discounted cost linear quadratic regulator problems. The author of [31] proposed a model-free synchronous PI algorithm for linear nonzero-sum quadratic differential games by constructing a Q-function for each player.…”

Section: Introductionmentioning

confidence: 99%

“…Further, a cooperative Q-learning method was proposed to solve multi-agent differential graphical games in [47]. In fact, the Q-learning methods in [31,44,47] belongs to on-policy RL, which has the drawback of insufficient exploration to the state space. Thus, to ensure the convergence to the Nash equilibrium, one should inject probing noises into the controllers to meet certain PE conditions, however, the effect of the probing noises on convergence has not been analyzed in [31,50].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Policy iteration based Q-learning for linear nonzero-sum quadratic differential games

Peng

Liang

et al. 2019

Sci. China Inf. Sci.

View full text Add to dashboard Cite

In this paper, a policy iteration-based Q-learning algorithm is proposed to solve infinite horizon linear nonzero-sum quadratic differential games with completely unknown dynamics. The Q-learning algorithm, which employs off-policy reinforcement learning (RL), can learn the Nash equilibrium and the corresponding value functions online, using the data sets generated by behavior policies. First, we prove equivalence between the proposed off-policy Q-learning algorithm and an offline PI algorithm by selecting specific initially admissible polices that can be learned online. Then, the convergence of the off-policy Qlearning algorithm is proved under a mild rank condition that can be easily met by injecting appropriate probing noises into behavior policies. The generated data sets can be repeatedly used during the learning process, which is computationally effective. The simulation results demonstrate the effectiveness of the proposed Q-learning algorithm.

show abstract

Section: Remark 4 Though the Initially Admissible Feedback Gainsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Policy iteration based Q-learning for linear nonzero-sum quadratic differential games

Peng

Liang

et al. 2019

Sci. China Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…When the state action space is small, the approximate optimal solution can be obtained using any RL method. When the space is large or continuous, RL does not converge to the optimal solution, which is called the “curse” of dimensionality. There are two problems when RL is applied in the robust control: RL overestimates the value functions using the minimum operator in the update rule.…”

Section: Introductionmentioning

confidence: 99%

Robust control under worst‐case uncertainty for unknown nonlinear systems using modified reinforcement learning

Perrusquía

2020

Intl J Robust & Nonlinear

View full text Add to dashboard Cite

Reinforcement learning (RL) is an effective method for the design of robust controllers of unknown nonlinear systems. Normal RLs for robust control, such as actor-critic (AC) algorithms, depend on the estimation accuracy. Uncertainty in the worst case requires a large state-action space, this causes overestimation and computational problems. In this article, the RL method is modified with the k-nearest neighbor and the double Q-learning algorithm. The modified RL does not need the neural estimator as AC and can stabilize the unknown nonlinear system under the worst-case uncertainty. The convergence property of the proposed RL method is analyzed. The simulations and the experimental results show that our modified RLs are much more robust compared with the classic controllers, such as the proportional-integral-derivative, the sliding mode, and the optimal linear quadratic regulator controllers. K E Y W O R D Sk-nearest neighbors, double estimator, overestimation, robust reward, state-action space, worst-case uncertainty INTRODUCTIONThe objective of robust control is to achieve robust performance in presence of disturbances. Most robust controllers are inspired by the optimal control theory, such as  2 control, 1,2 which minimizes a certain cost function to find an optimal controller. The most popular  2 controller is the linear quadratic regulator (LQR). 3 It does not work well in presence of disturbances. The  ∞ control can find a robust controller when the system has disturbances. Its performance is poor compared with the  2 control. 4 The combination of  2 and  ∞ , called  2 ∕ ∞ control, has both advantages, that is., it has optimal performance with bounded disturbances. 5 The  2 ∕ ∞ controller design needs a complete knowledge of the system dynamics. 6 These controllers are model-based. The time-varying quadratic optimization can be calculated by the zeroing neural network. It can simultaneously achieve the finite-time convergence and inherent noise tolerance. 7 However, it requires prefect activation functions. Model-free controllers, like the proportional-integral-derivative (PID) control, 8,9 the sliding mode control (SMC), 10 neural control, 11-16 among others, do not require dynamic knowledge of the system. However, parameter tuning and some prior knowledge of the disturbances prevent these model-free controllers to perform optimally.Reinforcement learning (RL) 17,18 is another effective method without models. It is designed in the sense of  2 control. 19 The temporal difference (TD) rule, such as Q-learning, is applied to find an optimal solution for Markov decision processes. 20 The advantage of RL over the other model-free methods is that it can reach optimal performances. Recent results show that RL methods can learn  2 and  ∞ controllers without system dynamics. 21 2920The main objective of RL for  2 and  ∞ is to minimize the total accumulative reward. For a robust controller, the reward is designed in the sense of control problems  2 and  ∞ . This robust reward can be static or dynam...

show abstract

“…The optimal control problems have been studied for more than half a century, and quite a number of methods have been presented, such as fuzzy logic theories [1][2][3][4] and approximation dynamic programming (ADP) algorithms. [5][6][7][8][9][10][11][12][13][14][15][16][17] With the rapid development of intelligent computation technologies in the last several decades, intelligent control methods have been applied in the analysis of nonlinear systems (see the works of Zhang et al, 18 Zhao et al, 19 and Huang and Chung 20 ). For nonlinear systems, the optimal control is to find the solutions to the Hamilton-Jacobi-Bellman (HJB) equation.…”

Section: Introductionmentioning

confidence: 99%

Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

Zhang

et al. 2017

Optim Control Appl Methods

View full text Add to dashboard Cite

Summary In this paper, a modified value iteration–based approximate dynamic programming method is proposed for a class of affine nonlinear continuous‐time systems, whose dynamics are partially unknown. The value iteration algorithm is established in an online fashion, and the convergence proof is given. To attenuate the effect caused by the unascertained characteristics of the system dynamics, the integral reinforcement learning scheme is also used. In the proposed approximate dynamic programming method, it is emphasized that the single‐network structure is utilized to estimate the value functions and the control policies. That is, the iteration process is implemented on the actor/critic structure, in which case only the critic NN is required to be identified. Then, the least‐squares scheme is derived for the NN weights updating. Finally, a linear system and a nonlinear system are tested to evaluate the performance of the proposed online value iteration algorithm. Both of the examples show the feasibility and effectiveness of the proposed algorithms.

show abstract

Continuous-Time Q-Learning for Infinite-Horizon Discounted Cost Linear Quadratic Regulator Problems

Cited by 83 publications

References 28 publications

Policy iteration based Q-learning for linear nonzero-sum quadratic differential games

Policy iteration based Q-learning for linear nonzero-sum quadratic differential games

Robust control under worst‐case uncertainty for unknown nonlinear systems using modified reinforcement learning

Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

Contact Info

Product

Resources

About