How Does the Number of Objective Function Evaluations Impact Our Understanding of Metaheuristics Behavior?

Abstract

Comparing various metaheuristics based on an equal number of objective function evaluations has become standard practice. Many contemporary publications use a specific number of objective function evaluations by the benchmarking sets definitions. Furthermore, many publications deal with the recurrent theme of late stagnation, which may lead to the impression that continuing the optimization process could be a waste of computational capabilities. But is it? Recently, many challenges, issues, and questions have been raised regarding fair comparisons and recommendations towards good practices for benchmarking metaheuristic algorithms. The aim of this work is not to compare the performance of several well-known algorithms but to investigate the issues that can appear in benchmarking and comparisons of metaheuristics performance (no matter what the problem is). This paper studies the impact of a higher evaluation number on a selection of metaheuristic algorithms. We examine the effect of a raised evaluation budget on overall performance, mean convergence, and population diversity of selected swarm algorithms and IEEE CEC competition winners. Even though the final impact varies based on current algorithm selection, it may significantly affect the final verdict of metaheuristics comparison. This work has picked an important benchmarking issue and made extensive analysis, resulting in conclusions and possible recommendations for users working with real engineering optimization problems or researching the metaheuristics algorithms. Especially nowadays, when metaheuristic algorithms are used for increasingly complex optimization problems, and meet machine learning in AutoML frameworks, we conclude that the objective function evaluation budget should be considered another vital optimization input variable.

How Much Do Swarm Intelligence and Evolutionary Algorithms Improve Over a Classical Heuristic From 1960?

“…Despite the importance of theoretical studies, empirical comparisons between evolutionary algorithms seem to be more popular than theoretical ones, even though they are always limited by the scale of problems, setting of comparison rules, and algorithms chosen for the competition. A relatively wide-scale comparison between up to 30 various evolutionary algorithms has been presented in numerous papers [29], [30], [31], [54], [55]. In addition, each year multiple novel algorithms compete in different competitions on Evolutionary Computations (e.g.…”

“…The impact of versatile other control parameters on the performance of specific kinds of Evolutionary Algorithms has also been addressed multiple times [64], [65], [66], [67]. As shown in numerous comparison papers, the performance of specific Evolutionary Algorithms would also depend on the number of allowed function calls [35], [55], [68], [69]. It is also known that the choice of the specific statistical test may affect the choice of the best algorithms [70], [71], [72], [73].…”

A Pose-Based Feature Fusion and Classification Framework for the Early Prediction of Cerebral Palsy in Infants

“…We present several plot representations of our Bayesian Optimisation in the supplementary information. In the plots, the number of function evaluations relates to the iteration number of the objective function, the min objective is the minimum value that the objective function has reached up to the current iteration, and the estimated minimum objectives are the mean values of the posterior distribution of the Gaussian process model of the objective function [25]. We also map the hyper-parameter variables to the classification performance metrics to determine the optimal hyper-parameters.…”

Philippine Eagle Optimization Algorithm

“…With the increasing number of nature-inspired algorithms, various benchmarking tests have been developed to examine their performance [46]. These include testing the algorithms on different types of functions [47], [48], and checking the number of objective function evaluations they use [49].…”