The impact of hyper-parameter tuning for landscape-aware performance regression and algorithm selection

Janković, Anja; Popovski, Gorjan; Eftimov, Tome; Doerr, Carola

doi:10.1145/3449639.3459406

Cited by 21 publications

(7 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance of our selectors may be improved by tuning the Random Forests or by using alternative regression techniques [11]. Nevertheless, even if the performances of the selectors increase, the possible gain is relatively low.…”

Section: Discussionmentioning

confidence: 99%

Automated Algorithm Selection for Radar Network Configuration

Renau,

Dreo,

Peres

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The configuration of radar networks is a complex problem that is often performed manually by experts with the help of a simulator. Different numbers and types of radars as well as different locations that the radars shall cover give rise to different instances of the radar configuration problem. The exact modeling of these instances is complex, as the quality of the configurations depends on a large number of parameters, on internal radar processing, and on the terrains on which the radars need to be placed. Classic optimization algorithms can therefore not be applied to this problem, and we rely on "trial-and-error" black-box approaches.In this paper, we study the performances of 13 black-box optimization algorithms on 153 radar network configuration problem instances. The algorithms perform considerably better than human experts. Their ranking, however, depends on the budget of configurations that can be evaluated and on the elevation profile of the location. We therefore also investigate automated algorithm selection approaches. Our results demonstrate that a pipeline that extracts instance features from the elevation of the terrain performs on par with the classical, far more expensive approach that extracts features from the objective function.

show abstract

Section: Discussionmentioning

confidence: 99%

Automated Algorithm Selection for Radar Network Configuration

Renau,

Dreo,

Peres

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Regression Models for Algorithm Performance Prediction. For the learning process, we considered random forest (RF) regression [2], as it provides promising results for algorithm performance prediction [12] and is one of the most commonly used algorithms for algorithm performance prediction studies in evolutionary computation. The RF algorithm was used as implemented by the scikit-learn package [20] in Python.…”

Section: Experimental Designmentioning

confidence: 99%

RF+clust for Leave-One-Problem-Out Performance Prediction

Nikolikj¹,

Doerr²,

Eftimov³

2023

Preprint

View full text Add to dashboard Cite

Per-instance automated algorithm configuration and selection are gaining significant moments in evolutionary computation in recent years. Two crucial, sometimes implicit, ingredients for these automated machine learning (AutoML) methods are 1) feature-based representations of the problem instances and 2) performance prediction methods that take the features as input to estimate how well a specific algorithm instance will perform on a given problem instance. Nonsurprisingly, common machine learning models fail to make predictions for instances whose feature-based representation is underrepresented or not covered in the training data, resulting in poor generalization ability of the models for problems not seen during training. In this work, we study leave-one-problem-out (LOPO) performance prediction. We analyze whether standard random forest (RF) model predictions can be improved by calibrating it with a weighted average of performance values obtained by the algorithm on problem instances that are sufficiently close to the problem for which a performance prediction is sought, measured by cosine similarity in feature space. While our RF+clust approach obtains more accurate performance prediction for several problems, its predictive power crucially depends on the chosen similarity threshold as well as on the feature portfolio for which the cosine similarity is measured, thereby opening a new angle for feature selection in a zero-shot learning setting, as LOPO is termed in machine learning.

show abstract

“…Here, we are learning Random Forest (RF) regression models where we also considered tuning part of the hyperparameters. Previous studies have already shown that RF can provide promising results dealing with automated performance prediction [17]. For hyperparameter tuning, we employed the grid search methodology, which performs an exhaustive search over a manually selected finite subset of candidate solutions of the hyperparameter space of the algorithm.…”

Section: Regression Model and Its Hyperparametersmentioning

confidence: 99%

The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variants

Kostovska¹,

Vermetten²,

Džeroski³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Selecting the most suitable algorithm and determining its hyperparameters for a given optimization problem is a challenging task. Accurately predicting how well a certain algorithm could solve the problem is hence desirable. Recent studies in single-objective numerical optimization show that supervised machine learning methods can predict algorithm performance using landscape features extracted from the problem instances.Existing approaches typically treat the algorithms as black-boxes, without consideration of their characteristics. To investigate in this work if a selection of landscape features that depends on algorithms' properties could further improve regression accuracy, we regard the modular CMA-ES framework and estimate how much each landscape feature contributes to the best algorithm performance regression models. Exploratory data analysis performed on this data indicate that the set of most relevant features does not depend on the configuration of individual modules, but the influence that these features have on regression accuracy does. In addition, we have shown that by using classifiers that take the features' relevance on the model accuracy, we are able to predict the status of individual modules in the CMA-ES configurations.

show abstract

The impact of hyper-parameter tuning for landscape-aware performance regression and algorithm selection

Cited by 21 publications

References 48 publications

Automated Algorithm Selection for Radar Network Configuration

Automated Algorithm Selection for Radar Network Configuration

RF+clust for Leave-One-Problem-Out Performance Prediction

The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variants

Contact Info

Product

Resources

About