A New Weighted Rank Correlation

Abstract

Additional information: Use policyThe full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-prot purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. Abstract: Problem Statement: There have been many cases in real life where two independent sources have ranked n objects, with the interest focused on agreement in the top rankings. Spearman's rho and Kendall's tau coefficients assigned equal weights to all rankings. As a result, the literature proposed several weighted correlation coefficients with emphasis on the top rankings, including the top-down, weighted Kendall's Tau and Blest's correlation coefficient. Approach: This article introduced a new weighted rank correlation coefficient that was sensitive to agreement in the top rankings. It presented the limiting distribution under the null hypothesis of independence and provided a summary of quantiles of the exact null distribution for n = 3(1)9. Results: The article summarized the power comparison between the new weighted coefficient and other weighted coefficients, and showed that the new weighted rank correlation coefficient provided the locally most powerful rank test. Conclusions/Recommendations: The new weighted correlation should be used along with other weighted coefficients when the interest focused on agreement in the top rankings, in order to make an effective inference.

New weighted rank correlation coefficients sensitive to agreement on top and bottom rankings

“…The weighted correlation coe cient proposed by Maturi and Abdelfattah [13] is sensitive to the agreement on the top (or bottom) rankings but not on the top and bottom simultaneously, also their rank correlation is not symmetric. Coolen-Maturi [3] extended the rank correlation in [13] for more than two sets of rankings but again the focus was only on the agreement on the top or bottom rankings. The new weighted rank correlation presented in Section 2.1, uses the weight scores in [13], is symmetric and can be used to evaluate the agreement on the top and the bottom rankings at the same time.…”

“…The first weighted rank correlation coe cient is based on the weight scores introduced by Maturi and Abdelfattah [13]. The weighted correlation coe cient proposed by Maturi and Abdelfattah [13] is sensitive to the agreement on the top (or bottom) rankings but not on the top and bottom simultaneously, also their rank correlation is not symmetric. Coolen-Maturi [3] extended the rank correlation in [13] for more than two sets of rankings but again the focus was only on the agreement on the top or bottom rankings.…”

“…Several correlation coe cients have been proposed in the literature which are more sensitive to the agreement on the top rankings, such as the top-down correlation coe cient by Iman and Conover [9] which is based on Savage scores, the weighted Kendall's tau by Shieh [17], the Blest's correlation coe cient [2] and its symmetric version [5]. Maturi and Abdelfattah [13] presented a weighted rank correlation coe cient with the flexibility of choosing weights to reflect the emphasis of agreement on the top rankings.…”

“…The first one is based on the weights that are proposed by Maturi and Abdelfattah [13] which is more sensitive to the agreement on the top and bottom rankings simultaneously. The second and the third are based on the order statistics and the quantiles of the Laplace distribution, respectively.…”

On Evaluation of Outlier Rankings and Outlier Scores

“…Weights, if used to balance classes, should be related to the class size. The traditional measures use weights solely based on ranks [30], being not or at least not directly related to the class sizes. Furthermore, evaluation in information retrieval heads towards a different goal where precision or recall are not meaningful to individuals seeking only a few hits.…”