Towards Feature-Based Performance Regression Using Trajectory Data

Abstract

Black-box optimization is a very active area of research, with many new algorithms being developed every year. This variety is needed, on the one hand, since different algorithms are most suitable for different types of optimization problems. But the variety also poses a meta-problem: which algorithm to choose for a given problem at hand? Past research has shown that per-instance algorithm selection based on exploratory landscape analysis (ELA) can be an efficient mean to tackle this meta-problem. Existing approaches, however, require the approximation of problem features based on a significant number of samples, which are typically selected through uniform sampling or Latin Hypercube Designs. The evaluation of these points is costly, and the benefit of an ELA-based algorithm selection over a default algorithm must therefore be significant in order to pay off. One could hope to by-pass the evaluations for the feature approximations by using the samples that a default algorithm would anyway perform, i.e., by using the points of the default algorithm's trajectory. We analyze in this paper how well such an approach can work. Concretely, we test how accurately trajectory-based ELA approaches can predict the final solution quality of the CMA-ES after a fixed budget of function evaluations. We observe that the loss of trajectory-based predictions can be surprisingly small compared to the classical global sampling approach, if the remaining budget for which solution quality shall be predicted is not too large. Feature selection, in contrast, did not show any advantage in our experiments and rather led to worsened prediction accuracy. The inclusion of state variables of CMA-ES only has a moderate effect on the prediction accuracy.

Towards exploratory landscape analysis for large-scale optimization

“…As in [4,7,39], we did not perform feature selection because it deteriorated the performance of a classifier in our preliminary experiment. A similar observation was reported in [14]. This may be due to a leave-one-problem-out cross-validation (LOPO-CV) [7,16] (see the next section), where the prediction model is validated on an unseen function.…”

Explainable Landscape Analysis in Automated Algorithm Performance Prediction

“…Previous studies have already shown that models trained to predict the target precision reached by an algorithm or its logarithmic value perform differently [8,9]. There are problem instances for which the model trained to predict the target precision works well, however there are also problem instances for which the model trained to predict the logarithmic value of the target precision works better.…”

“…Finally, in order to predict the performance of the algorithm on a new problem instance, a supervised ML method is trained using the landscape data as input data and the performance data as a target data. In recent studies, this is done by learning a single ML model using a set of ELA features that works well across all problem instances [8,9,12]. All these studies used classical feature selection methods to select the ELA features that should improve the performance of the ML model.…”

Explainable Landscape-Aware Optimization Performance Prediction

“…In [14], trajectory-based ELA regression models have been developed to predict the CMA-ES target precision obtained after a fixed number function evaluations. It has been shown that classical feature selection techniques applied on the ELA features in combination with random forest with no hyperparameter tuning did not lead to better performance results.…”

“…One issue in training those models is the selection of the ELA features that will be used to describe the problems. Some of the studies used the cheap ELA features, and some of them select a subset of the previously mentioned features using classical ML feature selection techniques [9], [14], [15]. The presented results showed that the selection of the ELA features portfolio influences the end performance prediction, but the impact of each feature to the end prediction result is still treated in a black-box manner.…”