Application of Deep Reinforcement Learning in Maneuver Planning of Beyond-Visual-Range Air Combat

Abstract

Beyond-visual-range (BVR) engagement becomes more and more popular in the modern air battlefield. The key and difficulty for pilots in the fight is maneuver planning, which reflects the tactical decision-making capacity of the both sides and determinates success or failure. In this paper, we propose an intelligent maneuver planning method for BVR combat with using an improved deep Q network (DQN). First, a basic combat environment builds, which mainly includes flight motion model, relative motion model and missile attack model. Then, we create a maneuver decision framework for agent interaction with the environment. Basic perceptive variables are constructed for agents to form continuous state space. Also, considering the threat of each side missile and the constraint of airfield, the reward function is designed for agents to training. Later, we introduce a training algorithm and propose perceptional situation layers and value fitting layers to replace policy network in DQN. Based on long short-term memory (LSTM) cell, the perceptional situation layer can convert basic state to high-dimensional perception situation. The fitting layer does well in mapping action. Finally, three combat scenarios are designed for agent training and testing. Simulation result shows the agent can avoid the threat of enemy and gather own advantages to threat the target. It also proves the models and methods of agents are valid and intelligent air combat can be realized.

Fighter Equipment Contribution Evaluation Based on Maneuver Decision

“…A comprehensive mathematical model of air combat confrontation is established to effectively improves the fidelity of confrontation state. Through a large number of machine-to-machine confrontation simulation experiments with various initial states based on the MATLAB platform [22,23], an analysis of the maneuvering decision process is performed; the results verify that the model can autonomously output reasonable maneuvers for a certain goal and ensure highquality results for fighter missile attack decisions and kill efficiency calculations. The proposed dynamic method based on influence diagrams can be used to realistically assess a situation involving BVR air combat, and the superiority and killing benefit calculations are more accurate than those based on static methods; therefore, this method can accurately relate equipment contributions and the roles of equipment in a system.…”

Research on Maneuvering Decision Algorithm Based on Improved Deep Deterministic Policy Gradient

“…However, when the space dimension increases, it faces the explosion problem of the state space and action space, and its decision accuracy is bound to be affected by fuzziness. Many scholars have adopted different DQN algorithms such as DQN [9,32], LSTM-DQN [33], and MS-DDQN [34] to realize the maneuver learning of UAV in short-range air combat and solved the decision problem of continuous state space. However, its action space basically adopts the form of maneuver action library, and the limited maneuver action library is difficult to reflect the maneuver action in actual air combat.…”

Autonomous maneuver decision-making method based on reinforcement learning and Monte Carlo tree search

“…However, the maneuver library of this method only contains five maneuvers, which cannot meet the needs of air combat. Hu et al ( 2021 ) proposed to use the improved deep Q network (Mnih et al, 2015 ) for maneuver decisions in autonomous air combat, constructed the relative motion model, missile attack model, maneuver decision-making framework, designed the reward function for training agents, and replaced the strategy network in deep Q network with the perception situation layer and value fitting layer. This method improves the winning rate of air combat, but the maneuver library is relatively simple and difficult to meet the needs of air combat.…”