Continuous Surrogate-Based Optimization Algorithms Are Well-Suited for Expensive Discrete Problems

Abstract

One method to solve expensive black-box optimization problems is to use a surrogate model that approximates the objective based on previous observed evaluations. The surrogate, which is cheaper to evaluate, is optimized instead to find an approximate solution to the original problem. In the case of discrete problems, recent research has revolved around discrete surrogate models that are specifically constructed to deal with these problems. A main motivation is that literature considers continuous methods, such as Bayesian optimization with Gaussian processes as the surrogate, to be sub-optimal (especially in higher dimensions) because they ignore the discrete structure by, e.g., rounding off real-valued solutions to integers. However, we claim that this is not true. In fact, we present empirical evidence showing that the use of continuous surrogate models displays competitive performance on a set of high-dimensional discrete benchmark problems, including a real-life application, against state-of-the-art discrete surrogatebased methods. Our experiments with different kinds of discrete decision variables and time constraints also give more insight into which algorithms work well on which type of problem.

Constrained Discrete Black-Box Optimization using Mixed-Integer Programming

“…In contrast, every feasible point of a binary problem lies on a corner of the 0-1 hypercube. A similar observation has been made in the context of IDONE (Bliek et al, 2021), which also uses a ReLU-based surrogate model: encoding the Rosenbrock problem using binary variables improves the performance of the IDONE algorithm, although the opposite happens for a Bayesian optimization algorithm (Karlsson et al, 2020).…”

Multidisciplinary design optimization with mixed categorical variables for aircraft design

“…Some other modelling strategies consist in computing a continuous model for each category [16], either by continuously relaxing the design variables [17], by using a multi-armed bandit strategy to handle the categorical choices [18] or by considering a Gower distance to model simultaneously the proximity over categorical and continuous variables [19]. Recently, a continuous relaxation BO based method [20] to tackle mixed integer variables has been shown to solve efficiently expensive-to-evaluate optimization problems. In fact, using continuous relaxation within BO leads to better results.…”

“…However, the relaxation of the categorical design variables increases the number of the hyperparameters needed (to be tuned) associated with the GP model. This in particular constrained the method in [20] to be used only for small dimensional optimization problems. Since, the construction of the GP model may not be scalable to practical applications involving a large number of mixed variables, some specific strategies have to be investigated to handle this complexity: the Kriging model with Partial Least Squares (KPLS) is one of the most used technique [21,22].…”

Bayesian Optimization for auto-tuning GPU kernels

“…Given the problems mentioned, it might seem odd to use a continuous surrogate model in the first place. However, as seen in Section III-B, discrete surrogate models have their drawbacks as well, and as shown by [42], continuous surrogate models can be well-suited for optimization of discrete problems.…”