Autonomous Control of Combat Unmanned Aerial Vehicles to Evade Surface-to-Air Missiles Using Deep Reinforcement Learning

Abstract

This paper proposes a new reinforcement learning approach for executing combat unmanned aerial vehicle (CUAV) missions. We consider missions with the following goals: guided missile avoidance, shortestpath flight and formation flight. For reinforcement learning, the representation of the current agent state is important. We propose a novel method of using the coordinates and angle of a CUAV to effectively represent its state. Furthermore, we develop a reinforcement learning algorithm with enhanced exploration through amplification of the imitation effect (AIE). This algorithm consists of self-imitation learning and random network distillation algorithms. We assert that these two algorithms complement each other and that combining them amplifies the imitation effect for exploration. Empirical results show that the proposed AIE approach is highly effective at finding a CUAV's shortest-flight path while avoiding enemy missiles. Test results confirm that with our method, a single CUAV reaches its target from its starting point 95% of the time and a squadron of four simultaneously operating CUAVs reaches the target 70% of the time. INDEX TERMSDeep reinforcement learning, combat unmanned aerial vehicle, deep learning, autonomous flight management system, path planning, exploration

Autonomous Unmanned Aerial Vehicle navigation using Reinforcement Learning: A systematic review

“…• Flock Centering: maintaining flight formation as suggested by Reynolds [95] involves three concepts: 1) flock centering, 2) avoiding obstacles, and 3) velocity matching. This topology was applied in several research papers [82,[96][97][98][99]]. • Leader-Follower Flocking: the flock leader has its mission of reaching destination, while the followers (other UAVs) flock with the leader with a mission of maintaining distance and relative position to the leader [100,101].…”

Learning user-defined sub-goals using memory editing in reinforcement learning

“…The final goal is for the agent to move from the starting point to the target point. I represented the coordinates of the agent as a state using an effective coordinate vector [33]. The action of the agent was set as a simple movement: left, right, up, and down.…”

A Fully Controllable Agent in the Path Planning using Goal-Conditioned Reinforcement Learning

“…Combining two optimization methods also has been studied [39]. Recently, with the development of deep learning, studies on the path planning using the RL have mainly been proposed [3], [6], [7], [9], [10], [11], [14], [15], [16], [17], [40], [41], [42]. They have supposed the specific scenario and set an environment to apply the agent in the path planning.…”

“…P ATH planning is a method to find an optimal route from the starting point to the target point. It has been widely used in various fields such as robotics [1], [2], [3], drone [4], [5], [6], [7], [8], [9], military service [10], [11], and self-driving car [12], [13]. Recently, reinforcement learning (RL) has been mainly studied for the path planning [3], [7], [9], [10], [11], [14], [15], [16], [17].…”

“…It has been widely used in various fields such as robotics [1], [2], [3], drone [4], [5], [6], [7], [8], [9], military service [10], [11], and self-driving car [12], [13]. Recently, reinforcement learning (RL) has been mainly studied for the path planning [3], [7], [9], [10], [11], [14], [15], [16], [17]. To get an optimal solution, it is essential to give enough reward for an agent to reach the goal and to set up a specific environment.…”