Autonomous Maneuver Decision of Air Combat Based on Simulated Operation Command and FRV-DDPG Algorithm

Abstract

With the improvement of UAV performance and intelligence in recent years, it is particularly important for unmanned aerial vehicles (UAVs) to improve the ability of autonomous air combat. Aiming to solve the problem of how to improve the autonomous air combat maneuver decision ability of UAVs so that it can be close to manual manipulation, this paper proposes an autonomous air combat maneuvering decision method based on the combination of simulated operation command and the final reward value deep deterministic policy gradient (FRV-DDPG) algorithm. Firstly, the six-degree-of-freedom (6-DOF) model is established based on the air combat process, UAV motion, and missile motion. Secondly, a prediction method based on the Particle swarm optimization radial basis function (PSO-RBF) is designed to simulate the operation command of the enemy aircraft, which makes the training process more realistic, and then an improved DDPG strategy is proposed, which returns the final reward value to the previous reward value in a certain proportion of time for offline training, which can improve the convergence speed of the algorithm. Finally, the effectiveness of the algorithm is verified by building a simulation environment. The simulation results show that the algorithm can improve the autonomous air combat maneuver decision-making ability of UAVs.

Hierarchical Reinforcement Learning Framework in Geographic Coordination for Air Combat Tactical Pursuit

“…real-world tasks, it is very challenging to design ideal reward functions that apply to all situations. To realize the model in this paper to adapt the reward functions of the three models [9,15,49], we mainly adjust the f(ϕ u ), f(ϕ t ), and f(D) in Equations ( 12) and (13).…”

“…In this experiment, we reproduce the models and algorithms in three papers [9,15,49], and apply the hierarchical reinforcement learning framework established in this paper to learn and train them, respectively, while mapping the reward functions shaped in the three papers in the corresponding sub-state spaces; then, in the air combat environment established in this paper, different models are compared in the same test scenarios, and the performance of the three original models is compared with that of the models after applying HRL. We use the benchmark performance comparison method proposed in Section 4.2 to compare the models proposed in the paper, as shown in Table 1.…”

“…Of these, confrontation behavior in air combat is complex and variable, with various modes [11], and it is difficult for traditional methods such as state machines and differential games to completely characterize the real-time decision-making state of pilots and devise further optimization according to different situations [12,13]. However, by modeling the air combat process as a Markov process [14], reinforcement learning methods can achieve continuous optimization of decision-making algorithms [15,16].…”

Methods for Identifying and Controlling Weak Links in the Supplier's Product Guarantee Capability

“…Product assurance is a refined quality management method. Through the application of system engineering method, we can grasp the key points and weak links in product development and production, implement the responsibilities of key positions, refine and decompose the quality management requirements, identify and control the risks in the whole process, and realize the whole process management and control of product quality [1].…”

UAV Air Game Maneuver Decision-Making Using Dueling Double Deep Q Network with Expert Experience Storage Mechanism

“…The shaping reward obtained by the inverse reinforcement learning algorithm for addressing reward sparsity effectively enhances air game strategy learning speed. To solve reinforcement learning algorithm convergence speed issues, Li et al [31] improved the deep deterministic policy gradient algorithm [32] (DDPG) convergence speed by proportionally returning the final reward value to other reward values in the same air game process. Targeting the DDPG exploration strategy's insufficient exploration and low data utilization, Wan et al [33] incorporated a heuristic exploration strategy to enhance the algorithm's exploration capacity.…”