SOLO: Search Online, Learn Offline for Combinatorial Optimization Problems

Abstract

We study combinatorial problems with real world applications such as machine scheduling, routing, and assignment. We propose a method that combines Reinforcement Learning (RL) and planning. This method can equally be applied to both the offline, as well as online, variants of the combinatorial problem, in which the problem components (e.g., jobs in scheduling problems) are not known in advance, but rather arrive during the decision-making process. Our solution is quite generic, scalable, and leverages distributional knowledge of the problem parameters. We frame the solution process as an MDP, and take a Deep Q-Learning approach wherein states are represented as graphs, thereby allowing our trained policies to deal with arbitrary changes in a principled manner. Though learned policies work well in expectation, small deviations can have substantial negative effects in combinatorial settings. We mitigate these drawbacks by employing our graph-convolutional policies as non-optimal heuristics in a compatible search algorithm, Monte Carlo Tree Search, to significantly improve overall performance. We demonstrate our method on two problems: Machine Scheduling and Capacitated Vehicle Routing. We show that our method outperforms custom-tailored mathematical solvers, state of the art learning-based algorithms, and common heuristics, both in computation time and performance.

A review on learning to solve combinatorial optimisation problems in manufacturing

“…In Wu et al. [159], the AM [9] was adapted together with some RNN units to solve the time‐dependent TSPTW variant. In Oren et al.…”

“…In Oren et al. [160], the deep Q‐learning was explored together with an improved Monte Carlo Tree Search (MCTS) to solve online CVRP. However, the problem scales of the aforementioned works remain rather small.…”

“…In Wu et al [159], the AM [9] was adapted together with some RNN units to solve the time-dependent TSPTW variant. In Oren et al [160], the deep Q-learning was explored together with an improved Monte Carlo Tree Search (MCTS) to solve online CVRP. However, the problem scales of the aforementioned works remain rather small.…”

“…However, for NP‐hard routing problems, constructing solutions can easily fall into the local minimum, which renders the needs of combining other post‐processing techniques, such as: (1) sampling (e.g. in refs [8, 9, 143, 144, 150, 152, 154–159, 167]) that repeats the stochastic construction process for multiple times, (2) beam search (e.g. in refs [148]) that performs an additional best‐first search process, (3) AS (e.g.…”

A Reinforcement Learning Approach for Scheduling Problems With Improved Generalization Through Order Swapping

“…Algorithms such as Taboo search [Taillard, 1994], simulated Annealing [Van Laarhoven et al, 1992], genetic algorithms and particle swarm optimization [Pezzella et al, 2008] have proven to solve the problem, but lack in either computation time or generalization capabilities. Advancements in DRL approaches in recent years have enabled considerable progress for the domain of COP applications [Cappart et al, 2021, Oren et al, 2021. Some of the major COPs have been successfully solved using DRL such as the Travelling Salesman Problem (TSP) [Zhang et al, 2021, d O Costa et al, 2020, Zhang et al, 2020b, the Knap Sack Problem [Afshar et al, 2020, Cappart et al, 2021 and the Steiner Tree Problem [Du et al, 2021].…”