The Reliability of Metrics Based on Graded Relevance

Abstract

NTCIR was the first large-scale IR evaluation conference to construct test collections with graded relevance assessments: the NTCIR-1 test collections from 1998 already featured relevant and partially relevant documents. In this paper, I first describe a few graded-relevance measures that originated from NTCIR (and a few variants) which are used across different NTCIR tasks. I then provide a survey on the use of graded relevance assessments and of graded relevance measures in the past NTCIR tasks which primarily tackled ranked retrieval. My survey shows that the majority of the past tasks fully utilised graded relevance by means of graded evaluation measures, but not all of them; interestingly, even a few relatively recent tasks chose to adhere to binary relevance measures. I conclude this paper by a summary of my survey in table form, and a brief discussion on what may lie beyond graded relevance.

Bootstrap-Based Comparisons of IR Metrics for Finding One Relevant Document

“…At AIRS 2005, Sakai [8,14] reported on stability and swap experiments that included IR metrics based on graded relevance, namely, Q-measure, R-measure and normalised (Discounted) Cumulative Gain (n(D)CG) [7]. But this study was limited to IR metrics for the task of finding as many relevant documents as possible.…”

“…As for the task of finding one highly relevant document, two previous studies based on the stability and the swap methods reported that P-measure [12] and O-measure [10] may be more stable and sensitive than RR. As Table 1 shows, all of these studies involving graded-relevance metrics [8,10,12,14] used either the NTCIR-3 or the NTCIR-5 data, but not both.…”

“…Next, we define Q-measure [8,14], which is very highly correlated with AveP but can handle graded relevance. Then, Q-measure is defined as:…”

“…In light of this, Sakai [10] proposed a metric called O-measure for the task of finding one highly relevant document. O-measure is a variant of Q-measure which is very highly correlated with Average Precision (AveP) but can handle graded relevance [8,14]. O-measure can also be regarded as a generalisation of RR (See Section 3).…”

Effective Mobile Search Using Element-based Retrieval

“…Evaluation measures for the iUnit Ranking subtask are nDCG k (k = 3, 5, 10, and 20) [1] and the Q-measure [21]. Metric normalized discounted cumulative gain (nDCG) is commonly used as a measure to evaluate document retrieval.…”