The aggregation paradox for statistical rankings and nonparametric tests

Abstract

The relationship between social choice aggregation rules and non-parametric statistical tests has been established for several cases. An outstanding, general question at this intersection is whether there exists a non-parametric test that is consistent upon aggregation of data sets (not subject to Yule-Simpson Aggregation Paradox reversals for any ordinal data). Inconsistency has been shown for several non-parametric tests, where the property bears fundamentally upon robustness (ambiguity) of non-parametric test (social choice) results. Using the binomial(n, p = 0.5) random variable CDF, we prove that aggregation of r(�2) constituent data sets-each rendering a qualitatively-equivalent sign test for matched pairs result-reinforces and strengthens constituent results (sign test consistency). Further, we prove that magnitude of sign test consistency strengthens in significance-level of constituent results (strong-form consistency). We then find preliminary evidence that sign test consistency is preserved for a generalized form of aggregation. Application data illustrate (in)consistency in non-parametric settings, and links with information aggregation mechanisms (as well as paradoxes thereof) are discussed.

Simpson's aggregation paradox in nonparametric statistical analysis: Theory, computation, and susceptibility in public health data

“…As RSS is a nonparametric form of scoring, only the order of elements influences the group ranking. We then aggregate the original data set with its ordinal replicate as in Nagaraja and Sanders [22] and consider whether (under what conditions) the pooled data yields a different group rank result under RSS than do its two constituent data sets. That is, we consider the conditions for strict Simpson Reversal, whereby the outcome in 1 (3) is obtained for each constituent data sequence, but outcome 3 (1) is obtained for the pooled sequence.…”

“…Similarly, Sanders et al [21] find that variation in outcome by aggregation rule is fairly common for rank sum scoring and other aggregation common aggregation rules. For nonparametric statistical testing, Nagaraja and Sanders [22] consider a case in which a data set is ordinally replicated and then pooled with the replicate data set. In such an environment, the authors prove that Simpson Reversals cannot occur if the sign test for matched pairs is applied to the primitive and pooled data sets.…”

“…Despite important theoretical contributions by Haunsperger and Saari [5] and Nagaraja and Sanders [22], there have been no computational studies that assess the incidence of Simpson Reversals in the case of nonparametric tests. Though we know the WMW Test yields instances of the Paradox, we cannot ascertain without computational support if these instances are fairly frequent, as in the case of path models, or somewhat rare, as in the case of contingency tables.…”