A Dimensional Comparison between Evolutionary Algorithm and Deep Reinforcement Learning Methodologies for Autonomous Surface Vehicles with Water Quality Sensors

Abstract

The monitoring of water resources using Autonomous Surface Vehicles with water-quality sensors has been a recent approach due to the advances in unmanned transportation technology. The Ypacaraí Lake, the biggest water resource in Paraguay, suffers from a major contamination problem because of cyanobacteria blooms. In order to supervise the blooms using these on-board sensor modules, a Non-Homogeneous Patrolling Problem (a NP-hard problem) must be solved in a feasible amount of time. A dimensionality study is addressed to compare the most common methodologies, Evolutionary Algorithm and Deep Reinforcement Learning, in different map scales and fleet sizes with changes in the environmental conditions. The results determined that Deep Q-Learning overcomes the evolutionary method in terms of sample-efficiency by 50–70% in higher resolutions. Furthermore, it reacts better than the Evolutionary Algorithm in high space-state actions. In contrast, the evolutionary approach shows a better efficiency in lower resolutions and needs fewer parameters to synthesize robust solutions. This study reveals that Deep Q-learning approaches exceed in efficiency for the Non-Homogeneous Patrolling Problem but with many hyper-parameters involved in the stability and convergence.

Censored deep reinforcement patrolling with information criterion for monitoring large water resources using Autonomous Surface Vehicles

“…A discrete formulation of the action space is chosen to reduce the number of possible action-state combinations. Discretizing the action space in this way has proven to be sufficient in previous works [5] and allows for better convergence of policies [35]. Thus, the ASV agent can choose from up to 8 possible actions that will result in a movement in 8 different directions with respect to a fixed reference system parallel to the axes of the navigation map M. The possible angles of direction Psi follow the bearings of a compass A :=[S, SE, E, NE, N, NW, W, SW].…”

An Informative Path Planner for a Swarm of ASVs Based on an Enhanced PSO with Gaussian Surrogate Model Components Intended for Water Monitoring Applications

“…In a more recent work [3], the authors work with multiple agents and the strategy applied was a centralized approach. In [14], the authors compare the performance of the Evolutionary Algorithm (EA) and deep reinforcement learning methodologies as monitoring systems. The results demonstrate the efficiency of the DRL technique under high-resolution conditions.…”

A rapid solution method for surface unmanned vessels motion planning problem based on optimal control

“…(1) Unmanned boat motion planning based on reinforcement learning has been a research hotspot in recent years [9] . This approach enables unmanned boats to learn the optimal motion strategy through autonomous interaction with the environment, continuously trying and learning from experience to achieve efficient and robust motion planning [10] . Riccardo [11] and others proposed a reinforcement learning method based on a physical model, which gradually improves control efficiency during the learning process to achieve tasks such as trajectory tracking.…”