An Optimized Path Planning Method for Coastal Ships Based on Improved DDPG and DP

Abstract

Deep Reinforcement Learning (DRL) is widely used in path planning with its powerful neural network fitting ability and learning ability. However, existing DRL-based methods use discrete action space and do not consider the impact of historical state information, resulting in the algorithm not being able to learn the optimal strategy to plan the path, and the planned path has arcs or too many corners, which does not meet the actual sailing requirements of the ship. In this paper, an optimized path planning method for coastal ships based on improved Deep Deterministic Policy Gradient (DDPG) and Douglas–Peucker (DP) algorithm is proposed. Firstly, Long Short-Term Memory (LSTM) is used to improve the network structure of DDPG, which uses the historical state information to approximate the current environmental state information, so that the predicted action is more accurate. On the other hand, the traditional reward function of DDPG may lead to low learning efficiency and convergence speed of the model. Hence, this paper improves the reward principle of traditional DDPG through the mainline reward function and auxiliary reward function, which not only helps to plan a better path for ship but also improves the convergence speed of the model. Secondly, aiming at the problem that too many turning points exist in the above-planned path which may increase the navigation risk, an improved DP algorithm is proposed to further optimize the planned path to make the final path more safe and economical. Finally, simulation experiments are carried out to verify the proposed method from the aspects of plan planning effect and convergence trend. Results show that the proposed method can plan safe and economic navigation paths and has good stability and convergence.

Movement Analytics: Current Status, Application to Manufacturing, and Future Prospects from an AI Perspective

“…Table 2 lists industrial applications in maritime industries [262,82], transportation [2,13,10,171,199,16], autonomous vehicles [263,228], health [2,178,10], behavior analysis [199,16], manufacturing [216,10,77] and indoor positioning [216,2].…”

“…The methodologies in the literature are categorized into four categories: data processing and mining, index measurement, causality analysis, and operational research. One of the most recent and relevant works in the maritime industry using movement analytics is by Du et al [82]. They worked on the problem of identifying the obstacles and optimize against those obstacles to determine the collision-free path for coastal ships with minimum turning points.…”

“…The maritime industry is another industry also benefited from the use of movement analytics [262,82]. Yang et al [262] focus on the comprehensive review of the literature regarding Automatic Information Systems (AIS) applications in maritime industries.…”

Network Architecture for Optimizing Deep Deterministic Policy Gradient Algorithms

“…Learning to tend to overestimate errors [25][26][27][28]. In Q-Learning, the update of the estimated value of an action by the learning algorithm is conducted by the ε-greedy policy y t � r + c max (Q(s t+1 , a t+1 )), hence the actual maximal value of an action is usually smaller than the estimated 2 Computational Intelligence and Neuroscience maximal value of this action as shown in the following equation:…”

Deep reinforcement learning based offloading decision algorithm for vehicular edge computing

“… Chen et al (2021) proposed to apply one-dimensional convolution and long short term memory network to DDPG algorithm to solve the problem of resource allocation in Chen et al (2021) . Du et al (2021) proposed to apply the long short term memory network to the DDPG algorithm to solve the problem of road planning and obtained good results.…”