Benchmarking evolutionary algorithms for single objective real-valued constrained optimization – A critical review

Abstract

Benchmarking plays an important role in the development of novel search algorithms as well as for the assessment and comparison of contemporary algorithmic ideas. This paper presents common principles that need to be taken into account when considering benchmarking problems for constrained optimization. Current benchmark environments for testing Evolutionary Algorithms are reviewed in the light of these principles. Along with this line, the reader is provided with an overview of the available problem domains in the field of constrained benchmarking. Hence, the review supports algorithms developers with information about the merits and demerits of the available frameworks.

Bio-inspired computation: Where we stand and what's next

“…Nowadays, in order to compare several algorithms it is crucial to ensure that improvement in results are not due to stochastic differences in runs. Thus, it is mandatory to apply statistical tests to clarify the statistical significance of the performance gaps found among algorithms [343]. However, the majority of benchmarks in Table 1 do not consider the application of statistical testing, neither in their experimental setting (the number of runs is lower than those recommended for them), nor in the comparison criterion established by the organizers.…”

On Selection of a Benchmark by Determining the Algorithms’ Qualities

“…These competitions include various types of benchmark problem suites, like: single-objective, large-scale, noisy, multiobjective, and constrained optimization. The CEC competitions provide specific test environments for detailed algorithm assessment and comparison [26]. Mainly, the test environment is especially popular for benchmarking the EAs.…”

How Does the Number of Objective Function Evaluations Impact Our Understanding of Metaheuristics Behavior?

“…These conditions can also be further advanced to limit the maximum iterations without improvement or reaching the desired objective function value [54]. Apart from the budget limitation, some benchmark recommendations also suggest additional termination condition of an error rate lower than a predefined threshold, for example, set to E −8 [55].…”

“…The performance measures often include the mean statistical error of the best, worst, mean and median solutions, and their standard deviations. However, since the No Free Lunch theorem [55] states, there is no "universal" best performing algorithm to solve any possible problem, solely performanceoriented experiments usually cannot lead to general assumptions. An utter win of one algorithm on one set of problems does not mean that the algorithm would be usable on a different set.…”