Target Search Control of AUV in Underwater Environment With Deep Reinforcement Learning

Abstract

The autonomous underwater vehicle (AUV) is widely used to search for unknown targets in the complex underwater environment. Due to the unpredictability of the underwater environment, this paper combines the traditional frontier exploration method with deep reinforcement learning (DRL) to enable the AUV to explore the unknown underwater environment autonomously. In this paper, a grid map of the search environment is built by the grid method. The designed asynchronous advantage actor-critic (A3C) network structure is used in the traditional frontier exploration method for target search tasks. This network structure enables the AUV to learn from its own experience and generate search strategies for the various unknown environment. At the same time, DRL and dual-stream Q-learning algorithms are used for AUV navigation to further optimize the search path. The simulations and experiments in an unknown underwater environment with different layouts show that the proposed algorithm can accomplish target search tasks with a high success rate, and it can adapt to different environments. In addition, compared to other search methods, the frontier exploration algorithm based on DRL can search a wider environment faster, which results in a higher search efficiency and reduced search time.INDEX TERMS Target search, frontier exploration, deep learning, reinforcement learning.

Research Progress of Path Planning Methods for Autonomous Underwater Vehicle

“…At the same time, DRL and dualstream Q-learning are applied to AUV obstacle avoidance and navigation to further optimize the search path. e simulation results show that the method can effectively control AUV to explore unknown environments [104].…”

“…It can improve the AUV's intelligence level and realize the path planning of the AUV in complex and unknown environments. In recent years, RL methods (such as Q-learning [99,100] and Sarsa [19]) or deep reinforcement learning methods [104] have achieved excellent results in AUV path planning. In 2018, Cao et al used reinforcement learning and Gaussian process regression to solve the path planning with bathymetric aids and modelled the value function as a Gaussian process to minimize the location uncertainty when the AUV reaches the target point [113].…”

Hunting Algorithm for Multi-AUV Based on Dynamic Prediction of Target Trajectory in 3D Underwater Environment

“…The training process uses the DQL algorithm. We have done a preliminary study on the DRL algorithm for path planning, and its specific working process is shown in [39].…”

“…They are represented by dots in different colors. The original position of the target is (39,41,11). It's represented by a purple star.…”

“…The simulation research target hunting when the velocity of AUV is slower than the speed of moving target. In the simulation with five AUVs, their original positions are (17,43,42), (39,45,3), (19,34,6), (40,22,5), and (37,9,23). The target is located in (44, 47, 38) originally and its trajectory is purple.…”