Reinforcement Learning Applications in Unmanned Vehicle Control: A Comprehensive Overview

Abstract

This paper briefly reviews the dynamics and the control architectures of unmanned vehicles; reinforcement learning (RL) in optimal control theory; and RL-based applications in unmanned vehicles. Nonlinearities and uncertainties in the dynamics of unmanned vehicles (e.g. aerial, underwater, and tailsitter vehicles) pose critical challenges to their control systems. Solving Hamilton–Jacobi–Bellman (HJB) equations to find optimal controllers becomes difficult in the presence of nonlinearities, uncertainties, and actuator faults. Therefore, RL-based approaches are widely used in unmanned vehicle systems to solve the HJB equations. To this end, they learn the optimal solutions by using online data measured along the system trajectories. This approach is very practical in partially or completely model-free optimal control design and optimal fault-tolerant control design for unmanned vehicle systems.

TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient

“…Recent years has witnessed dramatic progress of reinforcement learning (RL) and multi-agent reinforcement learning (MARL) in real life applications, such as unmanned vehicles (Liu et al 2022), traffic signal control (Noaeen et al 2022) and on-demand delivery (Wang et al 2023). Taking advantage of the centralized training decentralized execution (CTDE) (Oliehoek, Spaan, and Vlassis 2008; Kraemer and Banerjee 2016) paradigm, current cooperative MARL methods (Du et al 2023;Wang et al 2020a,b;Peng et al 2021;Zhang et al 2021;Zhou, Lan, and Aggarwal 2022) adopt value function factorization or a centralized critic to provide centralized learning signals to promote cooperation and achieve implicit or explicit credit assignment.…”

On Optimized Cooperative Control for Stochastic Nonlinear Multiagent Systems Using Switched-triggered Input

“…Optimal control [13] is crucial in the modern control field because it can determine the best operating scheme for a system under given constraints, thus achieving optimal system performance. There are numerous applications of optimal control across different domains, such as aerospace [14], automotive industry [15], and unmanned aerial [16], playing a significant role in enhancing system performance, reducing energy consumption, and optimizing resource utilization. In nonlinear systems, adaptive dynamic programming and reinforcement learning (RL) technologies play vital roles ( [17][18][19][20][21][22][23]).…”

Finite‐horizon Q‐learning for discrete‐time zero‐sum games with application to H∞$$ {H}_{\infty } $$ control

“…In order to deal with unknown dynamics, reinforcement learning (RL) [5][6][7] and approximate dynamic programming (ADP) [8][9][10] have been developed. Recent applica-tions of RL can be found in areas such as autonomous vehicles and unmanned aerial vehicles [11,12]. In RL methods, solving to the Bellman optimality equation typically requires function approximation techniques.…”