Bayesian inference for algorithm ranking analysis

Abstract

The statistical assessment of the empirical comparison of algorithms is an essential step in heuristic optimization. Classically, researchers have relied on the use of statistical tests. However, recently, concerns about their use have arisen and, in many fields, other (Bayesian) alternatives are being considered. For a proper analysis, different aspects should be considered. In this work we focus on the question: what is the probability of a given algorithm being the best? To tackle this question, we propose a Bayesian analysis based on the Plackett-Luce model over rankings that allows several algorithms to be considered at the same time.

Bayesian performance analysis for black-box optimization benchmarking

“…One approach to address this question is to explicitly model the ranking of the algorithms. In [5] a first attempt to answer that question was made by proposing a Bayesian Plackett-Luce model to perform the comparative analysis of experimental results.…”

“…In what follows, we will briefly review the Bayesian Plackett-Luce model, for more details, the interested reader is referred to [5]. As in any Bayesian model, we identify the three distributions mentioned above: the likelihood function, the prior distribution, and the posterior distribution.…”

“…Let us assume that the instances at hand constitute a random sample from that statistical population. 5 Then, upon taking the average value, the underlying statistical population becomes the average results obtained by the algorithms in a random instance from the population of instances. Conversely, if all values are accounted for, the underlying population becomes the result of the algorithms in random runs over random instances (again, having exactly the same number of repetitions for each instance is not a proper sample, but the results may be close enough to those obtained with a proper sample).…”

“…We will illustrate the differences between these two approaches upon comparing the algorithms' performance over two subsets of the functions at n = 625, corresponding to an "easy" subset of 5 In practice, the instances are often not randomly sampled but rather selected based on some criteria. Formally, this is not a correct way of proceeding if the goal is doing inference.…”

Recent trends in the use of statistical tests for comparing swarm and evolutionary computing algorithms: Practical guidelines and a critical review

“…[59] Frequentist -Optimisation Application of Deep Statistical Comparison of Multi-Objective Optimisation algorithms for an ensemble of quality indicators. [60] Bayesian Bayesian analysis based on a model over the algorithms' rankings. [61] Parametric -Classification Methodology for the definition of the sample sizes.…”