Aggregated statistical rankings are arbitrary

Abstract

In many areas of mathematics, statistics, and the social sciences, the intriguing, and somewhat unsettling, paradox occurs where the “parts” may give rise to a common decision, but the aggregate of those parts, the “whole”, gives rise to a different decision. The Kruskal-Wallis nonparametric statistical test on n samples which can be used to rank-order a list of alternatives is subject to such a Simpson-like paradox of aggregation. That is, two or more data sets each may individually support a certain ordering of the samples under Kruskal-Wallis, yet their union, or aggregate, yields a different outcome. An analysis of this phenomenon yields a computable criterion which characterizes which matrices of ranked data, when aggregated, can give rise to such a paradox. Copyright Springer-Verlag Berlin Heidelberg 2003

Combinatorics and Statistical Issues Related to the Kruskal–Wallis Statistic

“…In particular, data could be row-ordered (Haunsperger (2003) Table 7: z-scores for extreme distribution of ranks Greenwell (2011)), or both row-and column-ordered (Frame et al (1954), Griffiths and Lord (2011)). In this section of the paper, we discuss each of these data structures separately.…”

Statistical Significance of Ranking Paradoxes

“…In this setting, one could analyze the data separately or consider incorporating it into one data set before doing the analyses. The Simpson-like paradox described in Bargagliotti (submitted manuscript, 2008) and Haunsperger (2003) occurs when the analysis of each of the separate data sets leads to one rank-ordering of the alternatives but the analysis of the complete data leads to another rank-ordering. Of course this paradox depends on the procedure used to analyze each of the data sets.…”

“…Depending on which procedure is used to analyze the ranks, different rank-orderings of the alternatives may occur (Bargagliotti and Saari, submitted manuscript, 2008). Particularly interesting types of inconsistencies are Simpson-like paradoxes, in which the individual data sets give rise to one overall ranking, but the aggregate of the data sets gives rise to a different ranking (Haunsperger and Saari 1991;Haunsperger 1992;Haunsperger 1996;Haunsperger 2003;Bargagliotti, submitted manuscript, 2008). Haunsperger (2003) has shown this paradox exists when the Kruskal-Wallis nonparametric statistical procedure on n samples is used to rank-order a list of alternatives.…”

“…Particularly interesting types of inconsistencies are Simpson-like paradoxes, in which the individual data sets give rise to one overall ranking, but the aggregate of the data sets gives rise to a different ranking (Haunsperger and Saari 1991;Haunsperger 1992;Haunsperger 1996;Haunsperger 2003;Bargagliotti, submitted manuscript, 2008). Haunsperger (2003) has shown this paradox exists when the Kruskal-Wallis nonparametric statistical procedure on n samples is used to rank-order a list of alternatives. Bargagliotti (submitted manuscript, 2008) has shown that these paradoxes also occur when the Mann-Whitney (Mann and Whitney 1947) and Bhapkar's V (Bhapkar 1961) procedures are used.…”

Decomposition behavior in aggregated data sets

“…Finally, we recall Haunsperger's aggregation definitions (see [11]). For a given statistical procedure whose outcome is ranking (possibly with ties) of the candidates, and for all matrices of ranks:…”