Surrogate Assisted Feature Computation for Continuous Problems

Abstract

A possible approach to Algorithm Selection and Configuration for continuous black box optimization problems relies on problem features, computed from a set of evaluated sample points. However, the computation of the features proposed in the literature require a rather large number of such sample points, unlikely to be practical for expensive real-world problems. On the other hand, surrogate models have been proposed to tackle the optimization of expensive objective function. It is proposed in this paper to use surrogate models to approximate the values of the features at reasonable computational cost. Two experimental studies are conducted, using the well-known BBOB framework as testbench. First, the effect of sub-sampling is analyzed. Then, a methodology to compute approximate values for the features using a surrogate model is proposed, and validated from the point of view of retrieving BBOB classes. It is shown that when only small computational budgets are available, using surrogate models as proxies to compute the features can be beneficial.

Per instance algorithm configuration of CMA-ES with limited budget

“…However, all aforementioned works used a large sample size (≥ 500 × d) to compute features. And as expected, the error of computing the features increases the budget decreases, as demonstrated with a limited budget of ≤ 100 × d in [6,20]. This drawback can be somehow limited by building a surrogate model f from S and gathering further samples using f without additional evaluations of f [6,20].…”

“…The computation of features is the core element of the empirical performance model and its predictive power. Belkhir et al [6] had empirically shown that the accuracy of the feature critically depends on their computational costs. The present study was focused on low budgets to compute features.…”

“…The discriminative power of these features was assessed in the context of classification of optimization problems [6,20,[23][24][25], and of Algorithm Configuration [5,26].…”

“…The sample size n is set to k × d for some k ∈ N, and different values of k are be investigated (k ∈ {10, 30, 50}). For each k, two different approaches to the k × d budget for feature computations are also be investigated: the features can be computed directly from S (denoted ψ k ), or, following [6], a surrogate model f of f can be built from S and used as a proxi to compute some "surrogateassisted features" denoted ψ k . In the latter case, as in [6], a Random Forest regressor is used to learn f (using the Scikit-Learn library implementation with 20 trees and maximal a depth of 500).…”

“…First, for each of the 24 test functions from BBOB testbench, and for each dimensions listed in Section 6, 15 independent sample objective functions are generated and their respective features computed (some features can slightly differ for different sample of the same function, in accordance with Belkhir et al [6] that observed a higher in the features values when they are computed from small samples), and the best parameter setting of CMA-ES, as discovered by SMAC, are computed and their performance kept as the upper bound. The performance of the default CMA-Es setting is also computed, to be used as baseline results.…”