Rolling Cargo Management Using a Deep Reinforcement Learning Approach

Abstract

Loading and unloading rolling cargo in roll-on/roll-off are important and very recurrent operations in maritime logistics. In this paper, we apply state-of-the-art deep reinforcement learning algorithms to automate these operations in a complex and real environment. The objective is to teach an autonomous tug master to manage rolling cargo and perform loading and unloading operations while avoiding collisions with static and dynamic obstacles along the way. The artificial intelligence agent, representing the tug master, is trained and evaluated in a challenging environment based on the Unity3D learning framework, called the ML-Agents, and using proximal policy optimization. The agent is equipped with sensors for obstacle detection and is provided with real-time feedback from the environment thanks to its own reward function, allowing it to dynamically adapt its policies and navigation strategy. The performance evaluation shows that by choosing appropriate hyperparameters, the agents can successfully learn all required operations including lane-following, obstacle avoidance, and rolling cargo placement. This study also demonstrates the potential of intelligent autonomous systems to improve the performance and service quality of maritime transport.

BoxStacker: Deep Reinforcement Learning for 3D Bin Packing Problem in Virtual Environment of Logistics Systems

“…Recently, the bin packing problem has been solved through Deep Reinforcement Learning (DRL) [ 17 , 18 , 19 , 20 , 21 ]. Reinforcement Learning (RL) is a field of machine learning that entails a set of techniques for determining the optimum agent strategy and maximizing the reward of the agent [ 22 ]. DRL uses a deep neural network (DNN) to extend Reinforcement Learning without having to explicitly define the state space.…”

Optimal dual cycling operations in roll-on roll-off terminals

“…Man et al (2020) investigates methods for calculating the best and safest trajectories for loading large (out of gauge) cargo on RoRo ships without collision. In the same realm, Oucheikh et al (2021) proposes a deep reinforcement learning approach where artificial intelligence agents represent tug masters. The agents learn to navigate the environment of loading and unloading processes in a 3D-learning framework aiming at avoiding collisions with static as well as dynamic obstacles.…”

The Integration of Urban Freight in Public Transportation: A Systematic Literature Review

“…This suggests a gap in the literature that can be filled by pursuing simulation models to assess the impact of different packing and design strategies on efficiency and cost, as well as the effects of this integration on passenger transportation service quality. An interesting study was carried out by [71], where the authors aimed to teach an autonomous tug master to manage rolling cargo and perform loading and unloading operations. This training was performed through a deep reinforcement learning algorithm.…”