The executable experimental template pattern for the systematic comparison of metaheuristics

Object-Oriented Genetic Improvement for Improved Energy Consumption in Google Guava

“…Statistical testing was carried out using the Wilcoxon / Mann Whitney U Statistical Tests and Vargha-Delaney Effect size tests (as implemented in the Astraiea framework [10]). The results were obtained with 100 samples in each dataset.…”

Search-Based Energy Optimization of Some Ubiquitous Algorithms

“…In order to verify that the results are significant, we used the ASTRAIEA statistical framework [44] to perform the nonparametric Mann-Whitney U-test and the Vargha-Delaney Effect size test. The use of the latter is motivated by a trend within software engineering to augment significance tests for stochastic algorithms with effect size measures, with Vargha-Delaney being specifically recommended for this [45].…”

“…Statistical testing was carried out using the ASTRAIEA [44] Wilcoxon/Mann Whitney U and Vargha-Delaney Effect size tests. Results were obtained with 100 samples in each dataset.…”

Metaheuristics "In the Large"

“…giving the best solution found after a fixed number of iterations), but also continuously at each iteration 13 . Some recent work has also started to address the issue of fair comparison of algorithms by providing statistical testing frameworks which ensure that the preconditions for the various tests applied are actually met [75]. This is particularly important for metaheuristics, since common assumptions (e.g.…”

“…of normality) are not in general true. There is also a need to ground reported results in terms of "effect magnitude" [75]: for example, an improvement of 0.1% on the state-of-the-art may have more practical relevance for the Traveling Salesman Problem than for Bin-packing. In addition to statistical considerations, the specifics of the termination condition are obviously also a vital aspect of fair comparisons.…”