Black-box mixed-variable optimisation using a surrogate model that satisfies integer constraints

Abstract

A challenging problem in both engineering and computer science is that of minimising a function for which we have no mathematical formulation available, that is expensive to evaluate, and that contains continuous and integer variables, for example in automatic algorithm configuration. Surrogate-based algorithms are very suitable for this type of problem, but most existing techniques are designed with only continuous or only discrete variables in mind. Mixed-Variable ReLU-based Surrogate Modelling (MVRSM) is a surrogate-based algorithm that uses a linear combination of rectified linear units, defined in such a way that (local) optima satisfy the integer constraints. Unlike other methods, it also has a constant run-time per iteration. This method outperforms the state of the art on several synthetic benchmarks with up to 238 continuous and integer variables, and achieves competitive performance on two real-life benchmarks: XG-Boost hyperparameter tuning and Electrostatic Precipitator optimisation. CCS CONCEPTS• Mathematics of computing → Mathematical optimization; • Computing methodologies → Active learning settings; Search methodologies; • Theory of computation → Mixed discretecontinuous optimization.

CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization

“…The initial mean vector 𝒎 (0) is set to uniform random values in the range [1,3] for continuous variables and 0 for the binary variables, respectively. The covariance matrix and step-size are initialized with 𝑪 (0) = 𝑰 and 𝜎 (0) = 1, respectively.…”

“…The MI-BBO problems often appear in real-world applications such as, material design [11,17], topology optimization [4,16], placement optimization for CO 2 capture and storage [13], and hyper-parameter optimization of machine learning [9,10]. Several algorithms have been designed for MI-BBO so far, e.g., the extended evolution strategies [12] and surrogate model-based method [3]. However, despite the high demand for an efficient MI-BBO method, the BBO methods for mixed-integer problems are not actively developed compared to those for continuous or discrete problems.…”

A mixed-categorical correlation kernel for Gaussian process

“…In the next theorem, using the hypersphere decomposition properties [40], we will show that the correlation matrix R i , as given by (17), can be built in an equivalent way without the diagonal elements of the matrix Φ(Θ i ). Such result is of high interest as it reduces the number of hyperparameters from…”

“…Random forests are often used instead of GP as they also can model both mean and variance [15] and tree-structured Parzen estimators have been shown to be well-adapted for such problems [16]. Other surrogate models for blackbox include ReLU functions [17], piecewise linear neural network [18] or categorical regression splines [19]. Models other than GP could also be based on a mixed integer kernel as for support vector regression [20] or on a mixed integer distance as for radial basis functions [21].…”

A Survey on Sustainable Surrogate-Based Optimisation

“…Other open questions are concerned with: making efficient use of parallelisation, having multiple users in the PtI framework, including historic data in the SMBO framework, hyperparameter optimisation for surrogate learning, using smooth and/or interpretable surrogates, and general challenges in optimisation such as mixed variables, multimodality, nonlinearity and nonconvexity. Though many of these questions are gaining attention in recent SBO research, such as mixed variables [60,61], these open questions can serve as an incentive for SBO researchers to tackle such problems.…”