An Investigation into Prediction + Optimisation for the Knapsack Problem

Abstract

We study a prediction+optimisation formulation of the knapsack problem. The goal is to predict the profits of knapsack items based on historical data, and afterwards use these predictions to solve the knapsack. The key is that the item profits are not known beforehand and thus must be estimated, but the quality of the solution is evaluated with respect to the true profits. We formalise the problem, the goal of minimising expected regret and the learning problem, and investigate different machine learning approaches that are suitable for the optimisation problem. Recent methods for linear programs have incorporated the linear relaxation directly into the loss function. In contrast, we consider less intrusive techniques of changing the loss function, such as standard and multi-output regression, and learning-to-rank methods. We empirically compare the approaches on real-life energy price data and synthetic benchmarks, and investigate the merits of the different approaches.Combinatorial optimisation is crucial in today's society and used throughout many industries. In this paper, we work with the fundamental knapsack problem, which has been studied for over a century and is well understood [17,9]. It is studied in fields such as combinatorics, computer science, complexity theory, cryptography, and applied mathematics. It has numerous applications, including resource allocation problems where the aim is to select as many resources as possible under given financial constraints. The knapsack problem is NP-hard, though highly efficient solution methods exist for reasonably sized instances [9].In traditional optimisation, it is assumed that all parameters, e.g. the profits and weights in a knapsack, are precisely known beforehand. In practice, these are often crude estimates based on domain expertise or historic data. As we enter the age of big data, large amounts of data is available and thus parameters can be estimated with greater precision. For example, ongoing promotion and current weather might influence the demand. The question that arises is whether such contextual data, together with historical data, can be used to improve decision making, i.e. solve the underlying optimisation problem more effectively.Such problems are encountered in load shifting [10], where the aim is to create an energy-aware day-head schedule based on predicted hourly energy prices.

Learning Variable Activity Initialisation for Lazy Clause Generation Solvers

“…It is important to note that, similar to the approach proposed by Selsam and Bjørner [23] -and with works in the predict-and-optimise paradigm [6,8] -our ambition is not to achieve the best possible ML predictions. The reason for this is that more accurate predictions do not necessarily imply that they are more useful for the solver; rather the metric to optimise is the runtime of the solver.…”

Smart Predict-and-Optimize for Hard Combinatorial Optimization Problems

“…As a consequence, the ML models do not account for the optimization tasks (Wang et al 2006;Mukhopadhyay et al 2017). In recent years there is a growing interest in decision-focused learning (Elmachtoub and Grigas 2017;Demirović et al 2019;, that aims to couple ML and decision making.…”

“…Demirović et al (2019) investigate the predic-tion+optimisation problem for the knapsack problem, and prove that optimizing over predictions are as valid as stochastic optimisation over learned distributions, in case the predictions are used as weights in a linear objective. They further investigate possible learning approaches, and classified them into three groups: indirect approaches, which do not use knowledge of the optimisation problem; semi-direct approaches, which encode knowledge of the optimisation problem, such as the importance of ranking and direct approaches which encode or use the optimisation problem in the learning in some way (Demirović et al 2019). Our approach is a direct approach and we examine how to combine the best of such techniques in order to scale to large and hard combinatorial problems.…”

A Learning-Based Mathematical Programming Formulation for the Automatic Configuration of Optimization Solvers

“…A common issue in data-driven optimization is that using customary ML error metrics may not lead to good solutions of the optimization problem (see, for example, [14,17]). We tackled this issue by comparing the classical Mean Absolute Error, MAE S = i∈S |p i −p i |, where p i = p i (f, c) andp i =p i (f, c), to the custom metric cMAE S (δ) = i∈S loss i , where…”