Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments

Abstract

Autonomous motion planning (AMP) of unmanned aerial vehicles (UAVs) is aimed at enabling a UAV to safely fly to the target without human intervention. Recently, several emerging deep reinforcement learning (DRL) methods have been employed to address the AMP problem in some simplified environments, and these methods have yielded good results. This paper proposes a multiple experience pools (MEPs) framework leveraging human expert experiences for DRL to speed up the learning process. Based on the deep deterministic policy gradient (DDPG) algorithm, a MEP–DDPG algorithm was designed using model predictive control and simulated annealing to generate expert experiences. On applying this algorithm to a complex unknown simulation environment constructed based on the parameters of the real UAV, the training experiment results showed that the novel DRL algorithm resulted in a performance improvement exceeding 20% as compared with the state-of-the-art DDPG. The results of the experimental testing indicate that UAVs trained using MEP–DDPG can stably complete a variety of tasks in complex, unknown environments.

Trajectory Simulation Approach for Autonomous Vehicles Path Planning using Deep Reinforcement Learning

“…A state may contain an obstacle or not. Considering the idea is to simulate a real environment, which does not have a specific pattern of obstacle disposition, it is settled to randomly generate obstacles following some rules of degrees of difficulty, as it appears in [21]. Therefore, the RL model will be prepared and assessed in environments with degrees of difficulty: Easy; Medium; Difficult.…”

“…It is the module responsible for planning and (re) defining, in real-time, a route to be determined by the vehicle, from the mapping data, and transmitting this information to the Acting module. For this activity, artificial intelligence techniques, especially reinforcement learning, have been applied, as is the case in [21], in which models based on two Deep Reinforcement Learning techniques are presented called Deep Deterministic Policy Gradient (DDPG) and Multiple Experience Pools DDPG (MEP-DDPG) for planning autonomous movement of aerial unmanned vehicles (AUVs). Likewise [22] presents the application of DDPG for planning land vehicle routes.…”

A Smart Robotic Walker With Intelligent Close-Proximity Interaction Capabilities for Elderly Mobility Safety

“…For multi-robot motion control, a decentralized RL model is presented to learn a sensor-level collision avoidance policy in multi-robot systems (Fan et al, 2020). Domains like UAVs (Hu et al, 2020;Wan et al, 2020) and underwater vehicles (Carlucho et al, 2018;Chu et al, 2020) have also exploited RL for motion control for various purposes, e.g., robust flying, path planning and remote surveillance. In our work, we present a novel approach of exploiting mobility of the walker and RL techniques for efficient sound source localization (SSL).…”

Policy-Gradient and Actor-Critic Based State Representation Learning for Safe Driving of Autonomous Vehicles

“…Obtaining an optimal policy in reasonable time, taking decisions and actions under large state-spaces using DRL have been applied to network access, wireless caching, cognitive spectrum sensing, and network security. Some of the more recent DRL applications include modeling multiple experience pools for UAV autonomous motion planning in complex unknown environments [ 9 ], learning output reference model tracking for higher-order nonlinear systems with unknown dynamics [ 10 ], and pick and place operations in logistics using a mobile manipulator controlled with DRL [ 11 ]. The DRL paradigm has been extended to domains such as autonomous vehicles and has opened new research avenues [ 12 ].…”