Exploratory Landscape Analysis (ELA) subsumes a number of techniques employed to obtain knowledge about the properties of an unknown optimization problem, especially insofar as these properties are important for the performance of optimization algorithms. Where in a first attempt, one could rely on high-level features designed by experts, we approach the problem from a different angle here, namely by using relatively cheap low-level computer generated features. Interestingly, very few features are needed to separate the BBOB problem groups and also for relating a problem to high-level, expert designed features, paving the way for automatic algorithm selection.

No abstract

Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.

No abstract

The unrehearsed performance of music, called ‘sight-reading’ (SR), is a basic skill for all musicians. It is of particular interest for musical occupations such as the piano accompanist, the conductor, or the correpetiteur. However, up until now, there is no theory of SR which considers all relevant factors such as practice-related variables (e.g. expertise), speed of information processing (e.g. mental speed), or psychomotor speed (e.g. speed of trills). Despite the merits of expertise theory, there is no comprehensive model that can classify subjects into high- and low-performance groups. In contrast to previous studies, this study uses a data mining approach instead of regression analysis and tries to classify subjects into predetermined achievement classes. It is based on an extensive experiment in which the total SR performance of 52 piano students at a German music department was measured by use of an accompanying task. Additionally, subjects completed a set of psychological tests, such as tests of mental speed, reaction time, working memory, inner hearing, etc., which were found in earlier studies to be useful predictors of SR achievement. For the first time, classification methods (cluster analysis, regression trees, classification trees, linear discriminant analysis) were applied to determine combinations of variables for classification. Results of a linear discriminant analysis revealed a two-class solution with four predictors (cross-validated error: 15%) and a three-class solution with five predictors (cross-validated error: 33%).

No abstract

No abstract

We present methods to answer two basic questions that arise when benchmarking optimization algorithms. The first one is: which algorithm is the “best” one? and the second one is: which algorithm should I use for my real-world problem? Both are connected and neither is easy to answer. We present a theoretical framework for designing and analyzing the raw data of such benchmark experiments. This represents a first step in answering the aforementioned questions. The 2009 and 2010 BBOB benchmark results are analyzed by means of this framework and we derive insight regarding the answers to the two questions. Furthermore, we discuss how to properly aggregate rankings from algorithm evaluations on individual problems into a consensus, its theoretical background and which common pitfalls should be avoided. Finally, we address the grouping of test problems into sets with similar optimizer rankings and investigate whether these are reflected by already proposed test problem characteristics, finding that this is not always the case.