Reinforcement Learning-Based Tracking Control of USVs in Varying Operational Conditions

Abstract

We present a reinforcement learning-based (RL) control scheme for trajectory tracking of fully-actuated surface vessels. The proposed method learns online both a model-based feedforward controller, as well an optimizing feedback policy in order to follow a desired trajectory under the influence of environmental forces. The method's efficiency is evaluated via simulations and sea trials, with the unmanned surface vehicle (USV) ReVolt performing three different tracking tasks: The four corner DP test, straight-path tracking and curved-path tracking. The results demonstrate the method's ability to accomplish the control objectives and a good agreement between the performance achieved in the Revolt Digital Twin and the sea trials. Finally, we include an section with considerations about assurance for RL-based methods and where our approach stands in terms of the main challenges.

Estimating spatio-temporal fields through reinforcement learning

“…Second, system identification can sometimes be challenging in marine environments due to several factors such as unmodeled dynamics and environment’s unknowns; for that reason, model-free RL approaches can be an alternative in these scenarios. Examples of approaches that have used RL for ASVs or AUVs include path planning Yoo and Kim (2016) , control Cui et al (2017) and tracking Martinsen et al (2020) .…”

Reinforcement Learning based on Scenario-tree MPC for ASVs

“…As a result of imperfect models and uncertain disturbances, constraint satisfaction can in general not be guaranteed for MPC schemes. A common remedy, often denoted as scenario-tree MPC, is to discretize the underlying stochastic process, and describe the evolution of the uncertainty via a scenario tree [4], [5]. To that end, we consider a discretetime, constrained system with uncertain parameters θ:…”

“…Autonomous Surface Vehicles (ASVs) have been extensively investigated recently in industry and research [1], [2], [3]. However, designing control systems that can tackle obstacle avoidance and tracking control, with severe external time-varying disturbances due to the wind, wave, and ocean currents, is one of the most challenging research topics for ASVs in maritime engineering [4], [5]. In the control literature, the motion control scenarios of such vehicles are divided into target tracking, path following, path tracking, and path maneuvering [6].…”

COLREG-Compliant Collision Avoidance for Unmanned Surface Vehicle Using Deep Reinforcement Learning

“…In the field of robotics, DRL has been used to teach a humanoid robot to walk safely [34], to achieve multiagent tasks in the context of mobile robots [35], to learn complex reward functions by observing the behaviour of another robot [36], and for path smoothing and control tracking in robotic vehicle navigation [37]. In the context of underactuated surface vehicles, the methods have also demonstrated remarkable potential, yielding promising results in a multitude of studies, including [38]- [42] for the path following problems and [43]- [46] for the collision avoidance ones. However, to our knowledge, none of the previous works have demonstrated path following and collision avoidance utilising range finder sensors within a RL framework and further demonstrated it to work in a real environment using historical vessel trajectories obtained from automatic identification system (AIS) data.…”