Binary and graded relevance in IR evaluations—Comparison of the effects on ranking of IR systems

Kekäläinen, Jaana

doi:10.1016/j.ipm.2005.01.004

Cited by 75 publications

(50 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…that the top-heaviness curve for nCG (Järvelin and Kekäläinen 2002;Kekäläinen 2005), the undiscounted version of nDCG, is a completely flat line. That is, to nCG, it does not matter at all at which rank the relevant document is found.…”

Section: Rbpmentioning

confidence: 89%

On information retrieval metrics designed for evaluation with incomplete relevance assessments

Sakai¹,

Kando

2008

Inf Retrieval

103

View full text Add to dashboard Cite

Modern information retrieval (IR) test collections have grown in size, but the available manpower for relevance assessments has more or less remained constant. Hence, how to reliably evaluate and compare IR systems using incomplete relevance data, where many documents exist that were never examined by the relevance assessors, is receiving a lot of attention. This article compares the robustness of IR metrics to incomplete relevance assessments, using four different sets of graded-relevance test collections with submitted runs-the TREC 2003 and 2004 robust track data and the NTCIR-6 Japanese and Chinese IR data from the crosslingual task. Following previous work, we artificially reduce the original relevance data to simulate IR evaluation environments with extremely incomplete relevance data. We then investigate the effect of this reduction on discriminative power, which we define as the proportion of system pairs with a statistically significant difference for a given probability of Type I Error, and on Kendall's rank correlation, which reflects the overall resemblance of two system rankings according to two different metrics or two different relevance data sets. According to these experiments, Q 0 , nDCG 0 and AP 0 proposed by Sakai are superior to bpref proposed by Buckley and Voorhees and to Rank-Biased Precision proposed by Moffat and Zobel. We also point out some weaknesses of bpref and Rank-Biased Precision by examining their formal definitions.

show abstract

Section: Rbpmentioning

confidence: 89%

On information retrieval metrics designed for evaluation with incomplete relevance assessments

Sakai¹,

Kando

2008

Inf Retrieval

103

View full text Add to dashboard Cite

show abstract

“…DCG [10] is widely used to evaluate the quality of rankings. In order to compute DCG, we sort the items according to the Hamming distance between their binary codes to the binary codes for the user.…”

Section: Evaluation Metricsmentioning

confidence: 99%

Learning binary codes for collaborative filtering

Zhou

Zha

2012

Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

This paper tackles the efficiency problem of making recommendations in the context of large user and item spaces. In particular, we address the problem of learning binary codes for collaborative filtering, which enables us to efficiently make recommendations with time complexity that is independent of the total number of items. We propose to construct binary codes for users and items such that the preference of users over items can be accurately preserved by the Hamming distance between their respective binary codes. By using two loss functions measuring the degree of divergence between the training and predicted ratings, we formulate the problem of learning binary codes as a discrete optimization problem. Although this optimization problem is intractable in general, we develop effective relaxations that can be efficiently solved by existing methods. Moreover, we investigate two methods to obtain the binary codes from the relaxed solutions. Evaluations are conducted on three public-domain data sets and the results suggest that our proposed method outperforms several baseline alternatives.

show abstract

“…Voorhees [14] utilized nDCG in TREC Web Track evaluation weighting highly relevant documents by factors 1 to 1000 in relation to marginally relevant documents and she concluded that varying the gain function leads to different ranking of systems. In a similar study, Kekäläi-nen [11] also examined how different weighting schemes of relevance scores affect the ranking of systems and similarly to Voorhees [14] concluded that the larger the relative difference between relevance grades, the more the ranking of systems deviates from that in the binary case. Furthermore, by comparing the rankings of systems induced by the discounted cumulative gain (DCG) and cumulative gain without discounting (CG), she demonstrated that discounting the gain values also alters the induced ranking of systems.…”

Section: Introductionmentioning

confidence: 97%

Empirical justification of the gain and discount function for nDCG

Kanoulas

Aslam

2009

Proceedings of the 18th ACM Conference on Information and Knowledge Management

View full text Add to dashboard Cite

The nDCG measure has proven to be a popular measure of retrieval effectiveness utilizing graded relevance judgments. However, a number of different instantiations of nDCG exist, depending on the arbitrary definition of the gain and discount functions used (1) to dictate the relative value of documents of different relevance grades and (2) to weight the importance of gain values at different ranks, respectively.In this work we discuss how to empirically derive a gain and discount function that optimizes the efficiency or stability of nDCG. First, we describe a variance decomposition analysis framework and an optimization procedure utilized to find the efficiency-or stability-optimal gain and discount functions. Then we use TREC data sets to compare the optimal gain and discount functions to the ones that have appeared in the IR literature with respect to (a) the efficiency of the evaluation, (b) the induced ranking of systems, and (c) the discriminative power of the resulting nDCG measure.

show abstract

Binary and graded relevance in IR evaluations—Comparison of the effects on ranking of IR systems

Cited by 75 publications

References 21 publications

On information retrieval metrics designed for evaluation with incomplete relevance assessments

On information retrieval metrics designed for evaluation with incomplete relevance assessments

Learning binary codes for collaborative filtering

Empirical justification of the gain and discount function for nDCG

Contact Info

Product

Resources

About