Deep reinforcement learning-based collision avoidance for an autonomous ship

Chui

et al. 2021

Sensors

Real-time collision-avoidance navigation of autonomous ships is required by many application scenarios, such as carriage of goods by sea, search, and rescue. The collision avoidance algorithm is the core of autonomous navigation for Maritime autonomous surface ships (MASS). In order to realize real-time and free-collision under the condition of multi-ship encounter in an uncertain environment, a real-time collision avoidance framework is proposed using B-spline and optimal decoupling control. This framework takes advantage to handle the uncertain environment with limited sensing MASS which plans dynamically feasible, highly reliable, and safe feasible collision avoidance. First, owing to the collision risk assessment, a B-spline-based collision avoidance trajectory search (BCATS) algorithm is proposed to generate free-collision trajectories effectively. Second, a waypoint-based collision avoidance trajectory optimization is proposed with the path-speed decoupling control. Two benefits, a reduction of control cost and an improvement in the smoothness of the collision avoidance trajectory, are delivered. Finally, we conducted an experiment using the Electronic Chart System (ECS). The results reveal the robustness and real-time collision avoidance trajectory planned by the proposed collision avoidance system.

Section: Related Workmentioning

confidence: 99%

A Real-Time Collision Avoidance Framework of MASS Based on B-Spline and Optimal Decoupling Control

Zhang

Chui

et al. 2021

Sensors

“…For example, the artificial potential field method [9,10], the velocity obstacle algorithm [11,12], the dynamic window method [13], and the heuristic algorithm [14,15] have been widely used in research on how ships can avoid any collision. In the meantime, the advancement of artificial intelligence technology, particularly reinforcement learning, provides a new possibility for ships to avoid a collision due to its obvious superiority in problems that can be solved via decision making [16][17][18]. In the past few years, some typical ship intelligent collision avoidance models based on reinforcement learning methods have been proposed [7,19,20].…”

Section: Introductionmentioning

confidence: 99%

A COLREGs-Compliant Collision Avoidance Decision Approach Based on Deep Reinforcement Learning

Huang

Liu

et al. 2022

JMSE

It is crucial to develop a COLREGs-compliant intelligent collision avoidance system for the safety of unmanned ships during navigation. This paper proposes a collision avoidance decision approach based on the deep reinforcement learning method. A modified collision avoidance framework is developed that takes into consideration the characteristics of different encounter scenarios. Hierarchical reward functions are established to assign reward values to constrain the behavior of the agent. The collision avoidance actions of the agent under different encounter situations are evaluated on the basis of the COLREGs to ensure ship safety and compliance during navigation. The deep Q network algorithm is introduced to train the proposed collision avoidance decision framework, while various simulation experiments are performed to validate the developed collision avoidance model. Results indicate that the proposed method can effectively perform tasks that help ships avoid collisions in different encounter scenarios. The proposed approach is a novel attempt for intelligent collision avoidance decisions of unmanned ships.

“…The trained USV agent planned the path in a simulation marine environment with a large number of static obstacles existing successfully. Chun and Roh [ 7 ] proposed a new method for calculating the collision risk of ships using the ship domain and the closest point of approach (CPA). A fine collision avoidance path that follows the COLREGs was generated by the reinforcement learning proximal policy optimization (PPO) algorithm [ 8 ].…”

Section: Introductionmentioning

confidence: 99%

A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs

Fan

Sun

2022

Sensors

Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studied. Here, a reinforcement learning collision avoidance (RLCA) algorithm is proposed that complies with USV maneuverability. Notably, the reinforcement learning agent does not require any prior knowledge about USV collision avoidance from humans to learn collision avoidance motions well. The double-DQN method was used to reduce the overestimation of the action-value function. A dueling network architecture was adopted to clearly distinguish the difference between a great state and an excellent action. Aiming at the problem of agent exploration, a method based on the characteristics of USV collision avoidance, the category-based exploration method, can improve the exploration ability of the USV. Because a large number of turning behaviors in the early steps may affect the training, a method to discard some of the transitions was designed, which can improve the effectiveness of the algorithm. A finite Markov decision process (MDP) that conforms to the USVs’ maneuverability and COLREGs was used for the agent training. The RLCA algorithm was tested in a marine simulation environment in many different USV encounters, which showed a higher average reward. The RLCA algorithm bridged the divide between USV navigation status information and collision avoidance behavior, resulting in successfully planning a safe and economical path to the terminal.