Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization

Abstract

Combinatorial optimization has found applications in numerous fields, from aerospace to transportation planning and economics. The goal is to find an optimal solution among a finite set of possibilities. The well-known challenge one faces with combinatorial optimization is the state-space explosion problem: the number of possibilities grows exponentially with the problem size, which makes solving intractable for large problems. In the last years, deep reinforcement learning (DRL) has shown its promise for designing good heuristics dedicated to solve NP-hard combinatorial optimization problems. However, current approaches have two shortcomings: (1) they mainly focus on the standard travelling salesman problem and they cannot be easily extended to other problems, and (2) they only provide an approximate solution with no systematic ways to improve it or to prove optimality. In another context, constraint programming (CP) is a generic tool to solve combinatorial optimization problems. Based on a complete search procedure, it will always find the optimal solution if we allow an execution time large enough. A critical design choice, that makes CP non-trivial to use in practice, is the branching decision, directing how the search space is explored. In this work, we propose a general and hybrid approach, based on DRL and CP, for solving combinatorial optimization problems. The core of our approach is based on a dynamic programming formulation, that acts as a bridge between both techniques. We experimentally show that our solver is efficient to solve two challenging problems: the traveling salesman problem with time windows, and the 4-moments portfolio optimization problem. Results obtained show that the framework introduced outperforms the stand-alone RL and CP solutions, while being competitive with industrial solvers.Preprint. Under review.

Deep Policy Dynamic Programming for Vehicle Routing Problems

“…While most researches have focused on instances up to 100 nodes, some have attempted scaling to larger instances, which remains challenging (Ma et al, 2019;Fu et al, 2020). Related to our approach, Cappart et al (2020) propose to combine reinforcement learning, constraint programming and dynamic programming and experiment with the TSP with time windows. For surveys of machine learning for routing problems and combinatorial optimization in general, we refer to Mazyavkina et al (2020); Vesselinova et al (2020).…”

Unconstrained Binary Models of the Travelling Salesman Problem Variants for Quantum Optimization

“…The references [28,29] provide exact solutions to the problem using constraint programming. A more recent study combining constraint programming and reinforcement learning is presented in [30].…”

Learning to Solve Combinatorial Optimization Problems on Real-World Graphs in Linear Time

“…Pointer network [66] Supervised, Approximation GCN + Search [31] Supervised, Approximation Q-Learning + GNN [19] Model-free, Value-based Hierarchical RL + GAT [44] Model-free, Policy-based REINFORCE + LSTM with attention [47] Model-free, Policy-based REINFORCE + attention [20] Model-free, Policy-based RL + GAT [36] Model-free, Policy-based DDPG [23] Model-free, Policy-based REINFORCE + Pointer network [10] Model-free, Policy-based RL + NN [45] Model-free, Actor-Critic RL + GAT [14] Model-free, Actor-Critic AlphaZero: MCTS + GCN [51] Model-based, Given model Knapsack Problem REINFORCE + Pointer network [10] Model-free, Policy-based Bin Packing Problem (BPP) REINFORCE + LSTM [29] Model-free, Policy-based AlphaZero: MCTS + NN [38] Model-based, Given model Job Scheduling Problem (JSP) RL + LSTM [16] Model-free, Actor-Critic Vehicle Routing Problem (VRP) REINFORCE + LSTM with attention [47] Model-free, Policy-based RL + LSTM [16] Model-free, Policy-based RL + GAT [36] Model-free, Policy-based RL + NN [43] Model-free, Policy-based RL + GAT [25] Model-free, Actor-Critic Global Routing DQN + MLP [40] Model-free, Value-based Highest Safe Rung (HSR) AlphaZero: MCTS + CNN [71] Model-based, Given model Table 3: Classification of ML approaches for NP-hard combinatorial optimization by problem, method, and type.…”