BoxStacker: Deep Reinforcement Learning for 3D Bin Packing Problem in Virtual Environment of Logistics Systems

Abstract

Manufacturing systems need to be resilient and self-organizing to adapt to unexpected disruptions, such as product changes or rapid order, in supply chain changes while increasing the automation level of robotized logistics processes to cope with the lack of human experts. Deep Reinforcement Learning is a potential solution to solve more complex problems by introducing artificial neural networks in Reinforcement Learning. In this paper, a game engine was used for Deep Reinforcement Learning training, which allows visualization of view learning and result processes more intuitively than other tools, as well as a physical engine for a more realistic problem-solving environment. The present research demonstrates that a Deep Reinforcement Learning model can effectively address the real-time sequential 3D bin packing problem by utilizing a game engine to visualize the environment. The results indicate that this approach holds promise for tackling complex logistical challenges in dynamic settings.

Mixed Palletizing for Smart Warehouse Environments: Sustainability Review of Existing Methods

“…Reinforcement Learning is an effect technology that can solve combinatorial optimization problems [16]. Deep Reinforcement Learning can train algorithms for complex behavior using a reward function, which leads to better results when compared to legacy heuristic algorithms [22]. The recent advances in Artificial Intelligence allowed machine learning methods to also enter the domain of mixed palletizing through Reinforcement and deep Reinforcement Learning.…”

“…Such an innovative approach that effectively utilizes both a physics and a game engine for DRL is presented in [22]. Utilizing a game engine (like Unity3D) to visualize the 3D environment and a physics engine to conduct the training of the algorithm can lead to the automation of the loading process in such virtual scenarios.…”

“…A literature search was conducted on databases such as Google Scholar, Science Direct, arXiv, the Institute of Electrical and Electronics Engineers (IEEE), Multidisciplinary Digital Publishing Institute (MDPI), and Scopus. More than 200 scientific papers were searched and reviewed related to their relevance, in accordance with which this procedure appears under various names, like "palletising" [7][8][9], "palletizing" [10][11][12], "palletisation" [13,14], "palletization" [6,15,16], "pallet loading problem" [17][18][19], "pallet placement" [13,14,20], and "3D Bin Packing Problem" (pallet version as opposed to packing packages and loading containers) [21,22]. To address this ongoing issue, a typology was proposed in 1990 [23] and in 2007 [24] by researchers that concerns the area of Cutting and Packing problems.…”

Volumetric Techniques for Product Routing and Loading Optimisation in Industry 4.0: A Review

“…In two recent studies [11,47], researchers direct their focus towards tackling the 3D Container Loading Problem (3D-CLP) with DRL. Whereas Murdivien and Um [47] propose the application of DRL to a specific instance of the 3D-BPP using the Proximal Policy Optimisation (PPO) algorithm, and conduct experiments involving hyper-parameter adjustments, Que et al [11] introduce a groundbreaking approach, marked by a novel container state representation and a revised sequence of sub-actions within the DRL framework. Their DRL-based approach demonstrates superior performance in utilisation rate.…”

“…The Unity game engine is used by Murdivien and Um [47] as a pivotal platform for simulating and training a Deep Reinforcement Learning (DRL) model to automate the loading process in a simulated environment. The authors highlight Unity's advantages in terms of visualisation, user-friendly interface, and flexibility.…”