Online 3D Bin Packing with Constrained Deep Reinforcement Learning

Abstract

We solve a challenging yet practically useful variant of 3D Bin Packing Problem (3D-BPP). In our problem, the agent has limited information about the items to be packed into the bin, and an item must be packed immediately after its arrival without buffering or readjusting. The item's placement also subjects to the constraints of collision avoidance and physical stability. We formulate this online 3D-BPP as a constrained Markov decision process. To solve the problem, we propose an effective and easy-to-implement constrained deep reinforcement learning (DRL) method under the actor-critic framework. In particular, we introduce a feasibility predictor to predict the feasibility mask for the placement actions and use it to modulate the action probabilities output by the actor during training. Such supervisions and transformations to DRL facilitate the agent to learn feasible policies efficiently. Our method can also be generalized e.g., with the ability to handle lookahead or items with different orientations. We have conducted extensive evaluation showing that the learned policy significantly outperforms the state-of-the-art methods. A user study suggests that our method attains a human-level performance.

Online 3D Bin Packing Reinforcement Learning Solution with Buffer

“…However, it would be convenient to know the maximum score for evaluation purposes. For this reason, [2] constructs three types of sequences, CUT-1, CUT-2 and Random Sequence (RS). CUT-1 and CUT-2 items are first generated via cutting-stock, that is to say, a bin sized cuboid is randomly and recursively 'cut' until the sliced items match the size constraints.…”

“…All these three datasets consist of 2000 and 100 sequences for training and testing respectively. The item dimensions vary in the range between [2,5] in all three dimensions, forming a set of 64 different items, while the bin resolution is 10 × 10 × 10.…”

“…In our work, we opt for a more *This work is supported, in part, by "3D Bin Packing with Deep Reinforcement Learning" project funded by Hyundai Robotics Co. Ltd., in part, by "Edge Brain Based Intelligent Manufacturing" project IITP-2022-0-00067, in part, by AI Graduate School Program, Grant No.2019-0-00421, and by ICT Consilience Program, IITP-2020-0-01821, of the Institute of Information and Communication Technology Planning Evaluation (IITP), sponsored by the Korean Ministry of Science and Information Technology (MSIT). 1 Authors from the Artificial Intelligence School, Sungkyunkwan University, Suwon, South Korea, * Corresponding author: Sukhan Lee Lsh1@skku.edu practical definition based on [2], where the decisions are irreversible and items are delivered in sequence one by one (online), such that we give special attention to the immediate items B ⊂ I. In practice, a conveyor belt carries the item sequence to a robotic arm located at the head of the line.…”

“…[20], inspired from previously mentioned works, presented a solution for the 3D-BPP relying on Pointer Net. Another relevant work is presented in [2], the authors introduced an on-policy model-free DRL agent composed of an actor, a critic and a predictor, to predict action probabilities, value and feasibility mask respectively. Multiple on-policy and off-policy algorithms were trained, being the on-policy method [10] the one with the highest performance.…”

Learning to Design and Construct Bridge without Blueprint

“…The final architecture will be considered feasible only if all the blocks produce a flat surface and two cliffs are stably connected -fluctuations on the bridge surface or a tiny misplacement of blocks can result in a complete failure. Our task is much more challenging than similar assembly tasks like bin-packing, where a partial score can be achieved even if the bin is not fully packed [6], [7].…”