Judging competitions and benchmarks: a candidate election approach

Abstract

Machine learning progress relies on algorithm benchmarks. We study the problem of declaring a winner, or ranking "candidate" algorithms, based on results obtained by "judges" (scores on various tasks). Inspired by social science and game theory on fair elections, we compare various ranking functions, ranging from simple score averaging to Condorcet methods. We devise novel empirical criteria to assess the quality of ranking functions, including the generalization to new tasks and the stability under judge or candidate perturbation. We conduct an empirical comparison on the results of 5 competitions and benchmarks (one artificially generated). While prior theoretical analyses indicate that no single ranking function satisfies all desired properties, our empirical study reveals that the classical "average rank" method fares well. However, some pairwise comparison methods can get better empirical results.

Filtering participants improves generalization in competitions and benchmarks

“…We do not address here the process of generating a sound and reproducible ranking. However, many related problems have been investigated in the literature: the test set size needed to get good error rate estimations [4], score distributions for stochastic algorithms [10], the selection of the worst run to reduce chance in competitions [2], or the problem of fusing scores from multiple "judges" (multiple tasks and/or multiple metrics) [8].…”