Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectories

Abstract

Combinatorial optimisation problems framed as mixed integer linear programmes (MILPs) are ubiquitous across a range of real-world applications. The canonical branch-and-bound (B&B) algorithm seeks to exactly solve MILPs by constructing a search tree of increasingly constrained sub-problems. In practice, its solving time performance is dependent on heuristics, such as the choice of the next variable to constrain ('branching'). Recently, machine learning (ML) has emerged as a promising paradigm for branching. However, prior works have struggled to apply reinforcement learning (RL), citing sparse rewards, difficult exploration, and partial observability as significant challenges. Instead, leading ML methodologies resort to approximating high quality handcrafted heuristics with imitation learning (IL), which precludes the discovery of novel policies and requires expensive data labelling. In this work, we propose retro branching; a simple yet effective approach to RL for branching. By retrospectively deconstructing the search tree into multiple paths each contained within a sub-tree, we enable the agent to learn from shorter trajectories with more predictable next states. In experiments on four combinatorial tasks, our approach enables learning-to-branch without any expert guidance or pre-training. We outperform the current state-of-the-art RL branching algorithm by 3-5× and come within 20% of the best IL method's performance on MILPs with 500 constraints and 1000 variables, with ablations verifying that our retrospectively constructed trajectories are essential to achieving these results. * Work completed during internship at InstaDeep Preprint. Under review.

Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networks

“…Consequently, practitioners rely on either approximate algorithms, which give restricted performance guarantees and poor scalability (Williamson and Shmoys, 2011), or heuristics, which have limited solution efficacy (Halim and Ismail, 2019). Since the first application of neural networks to CO by Hopfield and Tank (1985), the last decade has seen a resurgence in ML-for-CO (Bello* et al, 2017;Dai et al, 2017;Barrett et al, 2019;Gasse et al, 2019;Barrett et al, 2022;Parsonson et al, 2022b). The advantages of ML-for-CO over approximation algorithms and heuristics include handling complex problems at scale, learning either without external input and achieving super-human performance or imitating strong but computationally expensive solvers, and (after training) leveraging the fast inference time of a DNN forward pass to rapidly generate solutions.…”

“…To select the algorithm hyperparameters, we conducted a Bayesian search across the search space summarised in Table 5, with simulations conducted in a light 32-worker RAMP environment with a maximum simulation run time of 2 × 10 5 seconds to speed up the search. We adopted similar search ranges to those used by Kurach et al (2019); Hoffman et al (2020); Parsonson et al (2022b). For each set of hyperparameters, we ran the algorithm for 100 learner steps (a.k.a.…”