Unmanned Aerial Vehicle Pitch Control under Delay Using Deep Reinforcement Learning with Continuous Action in Wind Tunnel Test

Abstract

Nonlinear flight controllers for fixed-wing unmanned aerial vehicles (UAVs) can potentially be developed using deep reinforcement learning. However, there is often a reality gap between the simulation models used to train these controllers and the real world. This study experimentally investigated the application of deep reinforcement learning to the pitch control of a UAV in wind tunnel tests, with a particular focus of investigating the effect of time delays on flight controller performance. Multiple neural networks were trained in simulation with different assumed time delays and then wind tunnel tested. The neural networks trained with shorter delays tended to be susceptible to delay in the real tests and produce fluctuating behaviour. The neural networks trained with longer delays behaved more conservatively and did not produce oscillations but suffered steady state errors under some conditions due to unmodeled frictional effects. These results highlight the importance of performing physical experiments to validate controller performance and how the training approach used with reinforcement learning needs to be robust to reality gaps between simulation and the real world.

Deep Deterministic Policy Gradient-Based Active Disturbance Rejection Controller for Quad-Rotor UAVs

“…In the future, the proposed controller will be used to conduct hardware-inthe-loop simulation experiments to further verify the stability of the algorithm. However, the gap between the simulation and the real world presents additional challenges, such as the oscillation of the controller [34]. DQN…”

Fixed-time trajectory tracking control of a quadrotor UAV under time-varying wind disturbances: theory and experimental validation

“…then, the disturbance observation error of observer can converge to 0 within a fixed time T 1 , and the convergence time is independent of the initial state. (18) where ε p > 0.…”

“…A sliding mode control method was discussed in [17], and finite-time convergence of the quadrotor trajectory tracking was proved by using Lyapunov theory. In addition, some learning-based control techniques have been implemented on the UAV, such as reinforcement learning [18] and deep learning [19,20].…”