Hierarchical Active Tracking Control for UAVs via Deep Reinforcement Learning

Abstract

Active tracking control is essential for UAVs to perform autonomous operations in GPS-denied environments. In the active tracking task, UAVs take high-dimensional raw images as input and execute motor actions to actively follow the dynamic target. Most research focuses on three-stage methods, which entail perception first, followed by high-level decision-making based on extracted spatial information of the dynamic target, and then UAV movement control, using a low-level dynamic controller. Perception methods based on deep neural networks are powerful but require considerable effort for manual ground truth labeling. Instead, we unify the perception and decision-making stages using a high-level controller and then leverage deep reinforcement learning to learn the mapping from raw images to the high-level action commands in the V-REP-based environment, where simulation data are infinite and inexpensive. This end-to-end method also has the advantages of a small parameter size and reduced effort requirements for parameter turning in the decision-making stage. The high-level controller, which has a novel architecture, explicitly encodes the spatial and temporal features of the dynamic target. Auxiliary segmentation and motion-in-depth losses are introduced to generate denser training signals for the high-level controller’s fast and stable training. The high-level controller and a conventional low-level PID controller constitute our hierarchical active tracking control framework for the UAVs’ active tracking task. Simulation experiments show that our controller trained with several augmentation techniques sufficiently generalizes dynamic targets with random appearances and velocities, and achieves significantly better performance, compared with three-stage methods.

E-VAT: An Asymmetric End-to-End Approach to Visual Active Exploration and Tracking

“…Nonetheless, the MAV dynamics are not fully exploited in the design of the control policy and the action space is discrete, which poses a hard limit on the achievable performance. A continuous action space is considered in [12], where a RL-based policy is coupled with a low-level PID control layer. However, the positioning of the MAV is constrained to a plane and thus the tracker is not free to move in 3D.…”

“…Model-based techniques (see, e.g., [23]) present design and integration issues that inherently limit their performance and entail tracking errors that may limit their applicability. On the other hand, existing learningbased approaches are affected by different constraints: (i) the target lies on a plane [22]; (ii) the tracker is controlled by discrete actions [11]; (iii) the agent is trained with continuous actions that are confined to a subset of the tracker action space [12]. To overcome these limitations, in this paper we provide the following contributions:…”

“…However, much less attention has been devoted to more complex platforms such as MAVs, which require a more sophisticated policy to be learned by the DRL agent. State-of-the-art (SotA) works have addressed this issue by relying on some simplifying assumptions, e.g., by ignoring the vehicle dynamics [11] or by constraining the possible control actions to a predefined subset of the action space [12]. Solutions based on these simplifications are, in general, less robust and performing.…”

A Reasonable Alternative System for Searching UAVs in the Local Area

“…In the event of a UAV failure or inability of the UAV to return to its "home" (initial) position, in the event of an accident or other loss (shooting down) of the UAV in such an area, it is essential to obtain to the location as soon as possible. Otherwise, the collected data that carry useful equipment, or even parts of the UAV, could be misused by the other party [8,9]. Since this is a specific, local area in which the UAV operates, it is possible to create alternative ways of tracing and detecting a particular target [10].…”

Efficiently Detecting Non-Stationary Opponents: A Bayesian Policy Reuse Approach under Partial Observability

“…They are usually combined with reinforcement learning (RL) and have achieved great success [6][7][8][9]. Many advanced deep RL algorithms have been proposed and used to solve complex decision-making problems such as game playing [9,10], robotics [11][12][13][14], and autonomous driving [15][16][17][18]. In multi-agent systems [19,20], if an opponent's policy is fixed, the agent can treat it as part of the stationary environment and learn the response policy using single-agent RL algorithms [21][22][23].…”