An Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay for Path Planning of Unmanned Surface Vehicles

Abstract

Unmanned Surface Vehicle (USV) has a broad application prospect and autonomous path planning as its crucial technology has developed into a hot research direction in the field of USV research. This paper proposes an Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay (IPD3QN) to address the slow and unstable convergence of traditional Deep Q Network (DQN) algorithms in autonomous path planning of USV. Firstly, we use the deep double Q-Network to decouple the selection and calculation of the target Q value action to eliminate overestimation. The prioritized experience replay method is adopted to extract experience samples from the experience replay unit, increase the utilization rate of actual samples, and accelerate the training speed of the neural network. Then, the neural network is optimized by introducing a dueling network structure. Finally, the soft update method is used to improve the stability of the algorithm, and the dynamic ϵ-greedy method is used to find the optimal strategy. The experiments are first conducted in the Open AI Gym test platform to pre-validate the algorithm for two classical control problems: the Cart pole and Mountain Car problems. The impact of algorithm hyperparameters on the model performance is analyzed in detail. The algorithm is then validated in the Maze environment. The comparative analysis of simulation experiments shows that IPD3QN has a significant improvement in learning performance regarding convergence speed and convergence stability compared with DQN, D3QN, PD2QN, PDQN, PD3QN. Also, USV can plan the optimal path according to the actual navigation environment with the IPD3QN algorithm.

Path Planning Algorithm for Unmanned Surface Vessel Based on Multiobjective Reinforcement Learning

“…The PER-DDQN improves the learning effect and the learning speed by introducing the DDQN and priority experience replay. Two Q -networks are used in DDQN to eliminate the bias caused by the greedy policy [ 34 ]. The current Q -network is used to calculate the action corresponding to the maximum Q -value, and the target Q -network is used to calculate the target Q -value corresponding to the maximum action.…”

ChemGymRL: A customizable interactive framework for reinforcement learning for digital chemistry

“…Though dueling DQN can show better performance than DQN, it is known to take vastly more training time and require larger networks as compared to DQN. 43 Similarly, several enhancements to experience replay techniques, such as prioritized experience replay 44 and hindsight experience replay, 45 could improve performance but take a longer training time as compared to DQN.…”

Artificial Intelligence in Marine Science and Engineering

“…Unmanned Surface Vehicle (USV) has a broad application prospect and autonomous path planning as its crucial technology has developed into a hot research direction in the field of USV research [3,4]. Paper [5] proposes an Improved Dueling Deep Double-Q Network Based on Prioritized Experience Replay (IPD3QN) to address the slow and unstable convergence of traditional Deep Q Network (DQN) algorithms in the autonomous path planning of USV. Firstly, the authors used the deep double Q-Network to decouple the selection and calculation of the target Q value action to eliminate overestimation.…”