Multiple UAVs Path Planning Based on Deep Reinforcement Learning in Communication Denial Environment

Abstract

In this paper, we propose a C51-Duel-IP (C51 Dueling DQN with Independent Policy) dynamic destination path-planning algorithm to solve the problem of autonomous navigation and avoidance of multiple Unmanned Aerial Vehicles (UAVs) in the communication denial environment. Our proposed algorithm expresses the Q function output by the Dueling network as a Q distribution, which improves the fitting ability of the Q value. We also extend the single-step temporal differential (TD) to the N-step timing differential, which solves the problem of inflexible updates of the single-step temporal differential. More importantly, we use an independent policy to achieve autonomous avoidance and navigation of multiple UAVs without any communication with each other. In the case of communication rejection, the independent policy can achieve the consistency of multiple UAVs and avoid the greedy behavior of UAVs. In multiple-UAV dynamic destination scenarios, our work includes path planning, taking off from different initial positions, and dynamic path planning, taking off from the same initial position. The hardware-in-the-loop (HITL) experiment results show that our C51-Duel-IP algorithm is much more robust and effective than the original Dueling-IP and DQN-IP algorithms in an urban simulation environment. Our independent policy algorithm has similar effects as the shared policy but with the significant advantage of running in a communication denial environment.

Energy Efficient Path Planning Scheme for Unmanned Aerial Vehicle Using Hybrid Generic Algorithm-Based Q-Learning Optimization

“…This can be described as a constraint that ensures that each point in the region of interest is a specific distance from the drone path. UAV must avoid collisions with barriers to comply with the obstacle avoidance limitation [36]. This can be described as a limitation that prevents the UAV's path from crossing any obstacles.…”

A Soft Actor-Critic Deep Reinforcement-Learning-Based Robot Navigation Method Using LiDAR

“…indicates the randomness of the policy and is calculated as shown in Equation (2). γ is the discount factor, which indicates the length of time in the future to be considered.…”

“…Deep reinforcement learning is an algorithm that integrates deep neural networks and reinforcement learning to solve complex decision-making tasks [2]. With the fitting ability of deep neural networks, it solves the mapping from observed features to strategy and value functions, and at the same time, it uses reinforcement-learning algorithms to define the optimization problem and optimization objective and continuously improves the decision-making ability of the agent in the process of interacting with information from the environment.…”

“…(1) We use fast path-generation algorithm to control the robot to generate expert trajectories, combine SAC reinforcement learning with imitation learning based on expert trajectories to solve the problem of the poor navigation ability of the agent in the initial state, and improve the training safety and convergence speed of reinforcement learning. (2) Considering that the priority of expert trajectory data is higher than that of agent trajectory data, we improve the priority calculation method on the basis of the TDerror (time-difference error) priority replay technique, which improves the utilization efficiency of the data in the experience replay pool. (3) We introduced an RNN to improve the obstacle-avoidance ability of SAC navigation policy in a dynamic environment.…”

A Stealth–Distance Dynamic Weight Deep Q-Network Algorithm for Three-Dimensional Path Planning of Unmanned Aerial Helicopter

“…As one of the DRL algorithms, the Deep Q-Network (DQN) algorithm is a method to approximate the Q-learning function through a neural network. DQN methods have been increasingly applied in the field of path planning, and several brilliant algorithms based on it have been put forward [22][23][24][25]. Yin Cheng et al [26] have developed a concise DRL obstacle-avoidance algorithm that designed a comprehensive reward function for behaviors such as obstacle avoidance, target approach, speed correction, and attitude correction in dynamic environments, using the deep Q-network (DQN) architecture, to overcome the usability issue caused by the complicated control law in the traditional analytic approach.…”