Best practices for comparing optimization algorithms

Abstract

The final publication is available at Springer via http://dx.doi.org/10.1007/s11081-017-9366-1 Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. We provide suggestions for each step of the comparison process and highlight the pitfalls to avoid when evaluating the performance of optimization algorithms. We also discuss various methods of reporting the benchmarking results. Finally, some suggestions for future research are presented to improve the current benchmarking process.

The Fast and the Robust: Trade‐Offs Between Optimization Robustness and Cost in the Calibration of Environmental Models

“…Multiple algorithm launches also provide a basis for appraising algorithm performance (e.g., Auger & Hansen, 2005;Tolson & Shoemaker, 2007;Costa et al, 2011;Arsenault et al, 2014;Beiranvand et al, 2017;Qin et al, 2017;Rardin & Uzsoy, 2001). Multiple launches yield the distributions of estimated optima and computational costs, and, importantly, indicate the frequency (empirical probability) with which the algorithm finds the global optimum.…”

“…Another question is whether the best algorithm is the one with the best average performance over multiple problems, or best median performance, or even the best minimax performance (here, meaning the least bad worst performance). All these aspects are inherently open ended (e.g., see some discussion in Costa et al, 2011;Beiranvand et al, 2017) and lead to the realm of multiobjective analysis (Belton & Stewart, 2002).…”

Optimal Reconfiguration of a Limited Parallel Robot for Forward Singularities Avoidance

“…The PilOPT algorithm presents the best performance. However, results greatly depend on the tuning of the specific parameters of each algorithm, e.g., the stopping conditions, population size or step sizes (Beiranvand et al, 2017). In fact, the main reason why the PilOPT algorithm outperforms the rest is that it only requires one parameter, which is the number of design evaluations determining when the algorithm stops, occurring when no improvement in the Pareto efficiency is observed.…”

The cyclic Douglas–Rachford algorithm with r-sets-Douglas–Rachford operators

“…The averaged runtimes are shown in Figure 2. The performance profiles comparing the methods, shown in Figure 3, were obtained as follows, see [22] and [9]. Let S denote the set of all 6 solvers compared (namely, the original 2-sets-DR scheme, and the cyclic r-sets-DR algorithm with r = 3, 5, 10, 20, 50).…”