HJB-RL: Initializing Reinforcement Learning with Optimal Control Policies Applied to Autonomous Drone Racing

Abstract

In this work we present a planning and control method for a quadrotor in an autonomous drone race. Our method combines the advantages of both model-based optimal control and model-free deep reinforcement learning. We consider a single drone racing on a track marked by a series of gates, through which it must maneuver in minimum time. Firstly we solve the discretized Hamilton-Jacobi-Bellman (HJB) equation to produce a closed-loop policy for a simplified, reduced order model of the drone. Next, we train a deep network policy in a supervised fashion to mimic the HJB policy. Finally, we further train this network using policy gradient reinforcement learning on the full drone dynamics model with a low-level feedback controller in the loop. This gives a deep network policy for controlling the drone to pass through a single gate. In a race course, this policy is applied successively to each new oncoming gate to guide the drone through the course. The resulting policy completes a highfidelity AirSim drone race with 12 gates in 34.89s (on average), outracing a model-based HJB policy by 33.20s, a supervised learning policy by 1.24s, and a trajectory planning policy by 12.99s, while a model-free RL policy was never able to complete the race.

Neural Optimal Control using Learned System Dynamics

“…Fig. 3: Trajectories generated by NeuralOC and our method for controlling a quadrotor to reach a desired state at [3,3,3].…”

“…We evaluate these methods on the task of controlling a quadrotor to reach a goal pose. The goal pose is set to the position [3,3,3] in an upright orientation. The initial state positions are sampled from a normal distribution around the origin N ([0, 0, 0], I), and the rest of the state variables are initialized as 0, corresponding to an upright orientation and no initial velocities.…”

“…Generating control signals in a global fashion is possible using the Hamilton-Jacobi-Bellman (HJB) equations, whose solution is the value function or the optimal cost-to-go. However, solving these equations is notoriously difficult and the state-of-the-art methods are grid-based, limiting their application in higher dimensional problems [2], [3]. Figure 1 illustrates a high-level comparison of these approaches.…”

Autonomous Drone Racing: Time-Optimal Spatial Iterative Learning Control within a Virtual Tube

“…The proposed spatial ILC is compared with [6] in the same racing environment to verify the proposed approach's modelfree and online fast iterative features. The racing environment, namely Soccer Field, is from [4] shown as Fig.…”

“…Given the conditions of the known environment, pushing drones to their physical limits presents challenges to researchers. There are also many existing solutions to autonomous competitions, including the use of continuous-time polynomial trajectory planning [5], the time-discrete trajectories method with reinforcement learning (RL) methods [6], [7], search and sampling-based methods [8], and modelbased optimization methods [9]. Continuous-time polynomial trajectory planning has high computational efficiency, but Shuli Lv, Yan Gao, Jiaxing Che, Quan Quan (Corresponding Author) are with School of Automation Science and Electrical Engineering, Beihang University, Beijing, 100191, P.R.…”