Branching improved Deep Q Networks for solving pursuit-evasion strategy solution of spacecraft

Abstract

<p style='text-indent:20px;'>With the continuous development of space rendezvous technology, more and more attention has been paid to the study of spacecraft orbital pursuit-evasion differential game. Therefore, we propose a pursuit-evasion game algorithm based on branching improved Deep Q Networks to obtain a space rendezvous strategy with non-cooperative target. Firstly, we transform the optimal control of space rendezvous between spacecraft and non-cooperative target into a survivable differential game problem. Next, in order to solve this game problem, we construct Nash equilibrium strategy and test its existence and uniqueness. Then, in order to avoid the dimensional disaster of Deep Q Networks in the continuous behavior space, we construct a TSK fuzzy inference model to represent the continuous space. Finally, in order to solve the complex and timeconsuming self-learning problem of discrete action sets, we improve Deep Q Networks algorithm, and propose a branching architecture with multiple groups of parallel neural Networks and shared decision modules. The simulation results show that the algorithm achieves the combination of optimal control and game theory, and further improves the learning ability of discrete behaviors. The algorithm has the comparative advantage of continuous space behavior decision, can effectively deal with the continuous space chase game problem, and provides a new idea for the solution of spacecraft orbit pursuit-evasion strategy.</p>

Pursuit Path Planning for Multiple Unmanned Ground Vehicles Based on Deep Reinforcement Learning

“…This paper focuses on DDQN [34]. This is because although DQN [35] is suitable for planning tasks, it suffers from the issue of q-value overestimation. To address this problem, DDQN employs two separate networks: the estimated network and the target network.…”

An Improved Approach towards Multi-Agent Pursuit–Evasion Game Decision-Making Using Deep Reinforcement Learning

“…Gu et al [18] presented an attention-based fault-tolerant model, which could also be applied to pursuit-evasion games, and the key idea was to utilize the multihead attention mechanism to select the correct and useful information for estimating the critics. To solve the complicated training problems caused by discrete action sets introduced by deep Q networks [19], Liu et al [20] transformed a space rendezvous optimization problem between a space vehicle and noncooperative target into a pursuit-evasion differential game. They introduced a branching architecture with a group of parallel neural networks and shared decision modules.…”

Two-Stage Pursuit Strategy for Incomplete-Information Impulsive Space Pursuit-Evasion Mission Using Reinforcement Learning

“…Different from the UAV PE game, the space PE game has a long mission duration and complex dynamics. In the field of space PE game, Liu [26] and Wang [27] developed the improved branching deep Q networks and the fuzzy actor-critic learning algorithm, respectively. These previous researches usually restricted the initial distance between the two spacecraft to reduce the PE game duration and used a simplified dynamical model to improve the computational efficiency.…”