Online 3D Bin Packing with Constrained Deep Reinforcement Learning

Abstract

We solve a challenging yet practically useful variant of 3D Bin Packing Problem (3D-BPP). In our problem, the agent has limited information about the items to be packed into a single bin, and an item must be packed immediately after its arrival without buffering or readjusting. The item's placement also subjects to the constraints of order dependence and physical stability. We formulate this online 3D-BPP as a constrained Markov decision process (CMDP). To solve the problem, we propose an effective and easy-to-implement constrained deep reinforcement learning (DRL) method under the actor-critic framework. In particular, we introduce a prediction-and-projection scheme: The agent first predicts a feasibility mask for the placement actions as an auxiliary task and then uses the mask to modulate the action probabilities output by the actor during training. Such supervision and projection facilitate the agent to learn feasible policies very efficiently. Our method can be easily extended to handle lookahead items, multi-bin packing, and item re-orienting. We have conducted extensive evaluation showing that the learned policy significantly outperforms the state-of-the-art methods. A preliminary user study even suggests that our method might attain a human-level performance.

A review on learning to solve combinatorial optimisation problems in manufacturing

“…This work also discusses multiple constraints and criteria that should be considered in online BPPs. Zhao et al discretise the bottom of a bin as the grid, with grid points to be selected for locating the front-leftbottom corner of a box, and optimise space utilisation rate in 3D-BPP by an on-policy actor-critic algorithm [21,109]. This method is limited to integer constrained box sizes and employ an additional neural network to predict infeasible grid points for pruning solution space.…”

“…This method is limited to integer constrained box sizes and employ an additional neural network to predict infeasible grid points for pruning solution space. To handle continuous sizes of boxes, the authors also propose a tree search-based learning method, where leaf nodes represent candidate placements for the next item, and they use attention-based neural network to select a leaf node and spread the tree [18]. Yang et al reflect the promising candidate actions (derived from heuristics) as MDP rewards for the agent, so as to guide the policy learning via PPO [107].…”

“…However, very a few current works learn to decide on all three aspects [20,66], since it will cause the issue of the large action space. This could be more intractable when continuous positions are considered [18]. From the perspective of DRL, such large action space could be potentially tackled by the algorithms for continuous action space, for example, DDPG, TD3, and decomposition techniques.…”

“…Despite very early attempts that couple machine learning (ML) with COPs [3][4][5], recent deep learning-based methods present promising results, which are comparable to classic optimisation methods [6][7][8]. Their success has inspired miscellaneous deep models to cope with specific COPs, for example, vehicle routing [9][10][11][12][13], scheduling [14][15][16][17], bin packing [18][19][20][21], or enhance performance in solving general COPs, for example, integer programming [22][23][24][25][26][27][28] and constraint programming [29][30][31]. The corresponding literature on the methods for specific COPs are reviewed respectively in refs [32][33][34][35].…”

Planning Irregular Object Packing via Hierarchical Reinforcement Learning

“…In order to efficiently generate the optimal sequence and placement of objects, heuristic methods [1], [3], [4] with the greedy objective minimize the object stack heights in the packing boxes. Since the greedy search results in sub-optimal solution and high computational cost for object arrangement, data-driven methods [2], [5], [6] employ the reinforcement learning framework for bin packing. However, the objects for packing in realistic applications are usually irregular.…”