Learning Improvement Heuristics for Solving Routing Problems

Abstract

Recent studies in using deep learning to solve the Travelling Salesman Problem (TSP) focus on construction heuristics, the solution of which may still be far from optimality. To improve solution quality, additional procedures such as sampling or beam search are required. However, they are still based on the same construction policy, which is less effective in refining a solution. In this paper, we propose to directly learn the improvement heuristics for solving TSP based on deep reinforcement learning. We first present a reinforcement learning formulation for the improvement heuristic, where the policy guides selection of the next solution. Then, we propose a deep architecture as the policy network based on self-attention. Extensive experiments show that, improvement policies learned by our approach yield better results than state-of-the-art methods, even from random initial solutions. Moreover, the learned policies are more effective than the traditional hand-crafted ones, and robust to different initial solutions with either high or poor quality.

Deep Policy Dynamic Programming for Vehicle Routing Problems

“…Most constructive neural methods are auto-regressive, predicting the next node given the partial tour constructed, but other works have considered predicting a 'heatmap' of promising edges at once (Nowak et al, 2017;Joshi et al, 2019a;Fu et al, 2020), which allows a tour to be constructed (using sampling or beam search) without further evaluating the model. Whereas these are constructive approaches, others have reported results with 'learning to search', where a neural network is used to guide a search procedure such as local search (Chen & Tian, 2019;Lu et al, 2020;Gao et al, 2020;Wu et al, 2019;Hottung & Tierney, 2019). While most researches have focused on instances up to 100 nodes, some have attempted scaling to larger instances, which remains challenging (Ma et al, 2019;Fu et al, 2020).…”

Learning Collaborative Policies to Solve NP-hard Routing Problems

“…Chen et al [29] proposed a DRL-based local search framework, termed NeuRewriter, that shows a promising performance on CVRP and job scheduling problems. Wu et al [30], and Costa et al [31] proposed a DRL-based TSP solver by learning the 2-opt. Their method improves the randomly generated solutions, unlike the method of Chen et al [29] rewrites a solution given by a conventional heuristic solver.…”

“…We follow baseline setting of Kool et al [12] and Costa et al [19]. We set DRL baselines including the S2V-DQN [11], EAN [23], GAT-T [30], DRL-2opt [19], and AM [12]. We show the results of S2V-DQN and EAN reported by Kool et al [12], and the results of GAT-T reported by Costa et al [19].…”

Guidelines for the Computational Testing of Machine Learning approaches to Vehicle Routing Problems

“…This process can then be emulated to generate (possibly smallersized) more representative CVRP instances, as it was done in Kool, Hoof, Gromicho, et al (2021) and in Hottung, Kwon, and Tierney (2021) (appendix). In Wu et al (2020), the authors directly used a subset of X instances along with distributions more commonly used in other ML works.…”

“…Currently, most of the proposed approaches aim at learning constructive heuristics, which sequentially extend a partial solution, possibly employing additional procedures such as sampling and beam search (see e.g., Bello et al (2017) and Hottung, Kwon, and Tierney (2021)). Few others, such as Wu et al (2020) and Chen and Tian (2019), instead, focus on learning improvement heuristics to guide the exploration of the search space and iteratively refine an existing solution.…”