COLREG-Compliant Collision Avoidance for Unmanned Surface Vehicle Using Deep Reinforcement Learning

Abstract

Path Following and Collision Avoidance, be it for unmanned surface vessels or other autonomous vehicles, are two fundamental guidance problems in robotics. For many decades, they have been subject to academic study, leading to a vast number of proposed approaches. However, they have mostly been treated as separate problems, and have typically relied on non-linear first-principles models with parameters that can only be determined experimentally. The rise of deep reinforcement learning in recent years suggests an alternative approach: end-to-end learning of the optimal guidance policy from scratch by means of a trial-and-error based approach. In this article, we explore the potential of Proximal Policy Optimization, a deep reinforcement learning algorithm with demonstrated state-of-the-art performance on Continuous Control tasks, when applied to the dual-objective problem of controlling an autonomous surface vehicle in a COLREGs compliant manner such that it follows an a priori known desired path while avoiding collisions with other vessels along the way. Based on high-fidelity elevation and AIS tracking data from the Trondheim Fjord, an inlet of the Norwegian sea, we evaluate the trained agent's performance in challenging, dynamic real-world scenarios where the ultimate success of the agent rests upon its ability to navigate nonuniform marine terrain while handling challenging, but realistic vessel encounters.

Deep Reinforcement Learning Controller for 3D Path Following and Collision Avoidance by Autonomous Underwater Vehicles

“…To find the right balance between penalizing being off-track and avoiding obstacles—which are competing objectives—the weight parameter is used to regulate the trade-off. This structure is adapted from the work by Meyer et al (2020a) ; Meyer et al (2020b) , which performed similar experiments in 2D. In addition, we add penalties to roll, roll rate, and the use of control actuation to form the complete reward function: …”

“…The execution layer facilitates the interaction between the deliberate and reactive architectures and decides the final commanded steering ( Tan, 2006 ). The hybrid approach is demonstrated in Meyer et al (2020a) where a DRL agent trained in a purely synthetic environment could achieve the combined objective of path following and collision avoidance with real sea traffic data (moving obstacles) in the Trondheim fjord while complying with collision avoidance regulations. There are still challenges in state-of-the-art COLAV methods for vehicles subjected to nonholonomic constraints, such as AUVs.…”

Hybrid analysis and modeling, eclecticism, and multifidelity computing toward digital twin revolution

“…One can investigate how the reinforcement learning (RL) agent trained with an environment with the coarse mesh behaves in the environment with fine resolution. These efforts can be built on our recent works in RL architectures [222,223,327] to devise new approaches for transfer learning from a low‐fidelity model to a high‐fidelity model. It would be a nice attempt to explore constructing new reward functions that are more suited to multi‐x environments to take into account the efficient sampling of big data.…”

“…Therefore, hybrid methods could be instrumental in discovering new control laws. For example, recently, we show how a RL agent can learn complicated control laws through trial and error to achieve complicated tasks of path following and collision avoidance simultaneously [222,223,327]. However, such generalizable learning happens in a black‐box manner so their applicability in critical applications is foreseen to be limited unless the learned control laws can be expressed in comprehensible mathematical form.…”

Comparing Deep Reinforcement Learning Algorithms’ Ability to Safely Navigate Challenging Waters

“…After applying the PPO algorithm in a stochastic, synthetic environment ( Meyer, 2020a ), found that the trained agent perfectly generalized to multiple real-world scenarios simulating trafficked areas in the Trondheim fjord, Norway. Meyer et al (2020a) expands on Meyer et al (2020b) by hand-crafting a reward function that encourages the RL agent to comply with the International Regulations for Preventing Collisions at Sea (COLREGs) using the PPO algorithm. Havenstrøm et al (2021) applies a curriculum learning technique with the PPO algorithm to control a 6-DOF underactuated autonomous underwater vehicle (AUV), gradually increasing the presence and severity of obstacles and disturbances during the RL training process.…”