Identifying minimal set of Exploratory Landscape Analysis features for reliable algorithm performance prediction

Nikolikj, Ana; Trajanov, Risto; Cenikj, Gjorgjina; Korošec, Peter; Eftimov, Tome

doi:10.1109/cec55065.2022.9870439

Cited by 4 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, the availability of instances allows for the evaluation of AutoML approaches, e.g., algorithm selection, by enabling a cross-validation approach across instances. This generally leads to better results than cross-validation on a function-level, since the functions themselves have been designed specifically to contain different challenges to the optimizer, so leaving a function out of the training set generally makes the transfer between train and test challenging [13,23].…”

Section: The Bbob Problem Suitementioning

confidence: 99%

“…However, the challenge with using BBOB for algorithm selection lies in the evaluation of the results. One method is a leave-one-function-out technique [23], which uses 23 functions for training and the remaining one for testing. This approach tends to show poor performance since each problem has been designed to represent different high-level challenges for the optimization algorithm.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using Affine Combinations of BBOB Problems for Performance Assessment

Vermetten

Doerr

2023

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

Choosing a set of benchmark problems is often a key component of any empirical evaluation of iterative optimization heuristics. In continuous, single-objective optimization, several sets of problems have become widespread, including the well-established BBOB suite. While this suite is designed to enable rigorous benchmarking, it is also commonly used for testing methods such as algorithm selection, which the suite was never designed around.We present the MA-BBOB function generator, which uses the BBOB suite as component functions in an affine combination. In this work, we describe the full procedure to create these affine combinations and highlight the trade-offs of several design decisions, specifically the choice to place the optimum uniformly at random in the domain. We then illustrate how this generator can be used to gain more low-level insight into the function landscapes through the use of exploratory landscape analysis.Finally, we show a potential use-case of MA-BBOB in generating a wide set of training and testing data for algorithm selectors.Using this setup, we show that the basic scheme of using a set of landscape features to predict the best algorithm does not lead to optimal results, and that an algorithm selector trained purely on the BBOB functions generalizes poorly to the affine combinations.CCS Concepts: • Theory of computation → Random search heuristics; Design and analysis of algorithms; Bio-inspired optimization.

show abstract

Section: The Bbob Problem Suitementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Using Affine Combinations of BBOB Problems for Performance Assessment

Vermetten

Doerr

2023

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

show abstract

“…Three randomly selected Differential Evolution (DE) configurations are included in the analysis. Their hyper-parameters are set as presented in [13]. The population size of the algorithm is set equal to the problem dimension (10).…”

Section: Experimental Designmentioning

confidence: 99%

“…The LOPO performance prediction can be stated as an ML task known as zero-shot learning [10]- [12]. Nikolikj et al introduced RF+clust for LOPO algorithm performance prediction [13]. The approach calibrates the prediction obtained by a Random Forest (RF) model [14] for a given test problem with a weighted mean of the algorithm performance on problems from the training data that are the most similar to the test problem, based on their feature representation.…”

Section: Introductionmentioning

confidence: 99%

Explaining Differential Evolution Performance Through Problem Landscape Characteristics

Nikolikj

Korošec

Eftimov

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Leave-one-problem-out (LOPO) performance prediction requires machine learning (ML) models to extrapolate algorithms' performance from a set of training problems to a previously unseen problem. LOPO is a very challenging task even for state-of-the-art approaches. Models that work well in the easier leave-one-instance-out scenario often fail to generalize well to the LOPO setting. To address the LOPO problem, recent work suggested enriching standard random forest (RF) performance regression models with a weighted average of algorithms' performance on training problems that are considered similar to a test problem. More precisely, in this RF+clust approach, the weights are chosen proportionally to the distances of the problems in some feature space. Here in this work, we extend the RF+clust approach by adjusting the distance-based weights with the importance of the features for performance regression. That is, instead of considering cosine distance in the feature space, we consider a weighted distance measure, with weights depending on the relevance of the feature for the regression model. Our empirical evaluation of the modified RF+clust approach on the CEC 2014 benchmark suite confirms its advantages over the naive distance measure. However, we also observe room for improvement, in particular with respect to more expressive feature portfolios.

show abstract