Integration of adaptive control and reinforcement learning for real-time control and learning

Abstract

This paper considers the problem of real-time control and learning in dynamic systems subjected to uncertainties. Adaptive approaches are proposed to address the problem, which are combined with methods and tools in Reinforcement Learning (RL) and Machine Learning (ML). Algorithms are proposed in continuous-time that combine adaptive approaches with RL leading to online control policies that guarantee stable behavior in the presence of parametric uncertainties that occur in real-time. Algorithms are proposed in discrete-time that combine adaptive approaches proposed for parameter and output estimation and ML approaches proposed for accelerated performance that guarantee stable estimation even in the presence of time-varying regressors, and for accelerated learning of the parameters with persistent excitation. Numerical validations of all algorithms are carried out using a quadrotor landing task on a moving platform and benchmark problems in ML. All results clearly point out the advantage of the proposed integrative approaches for real-time control and learning.

Accelerated Performance and Accelerated Learning with Discrete-Time High-Order Tuners

“…The high-order tuner in [6] is continuous, and is later proved in [5] to result in exponential parameter convergence. The same exponential convergence results have also been proved in [9] using a different approach from [5]. But a discrete-time parameter convergence analysis has been lacking in the literature.…”

Adaptive Control and Intersections with Reinforcement Learning

“…The fact that the two approaches are different suggests that there are ways in which they could be integrated so as to realize their combined advantages. The focus of AC on stability and RL on optimality suggests that one such candidate is a multiloop approach, with an inner loop focused on AC methods that are capable of delivering real-time performance with stability guarantees and an outer loop focused on RL methods that can anticipate an optimal policy when the sim-to-real gap is small (93)(94)(95)(96)(97).…”