Evaluating diversified search results using per-intent graded relevance

Abstract

Search queries are often ambiguous and/or underspecified. To accomodate different user needs, search result diversification has received attention in the past few years. Accordingly, several new metrics for evaluating diversification have been proposed, but their properties are little understood. We compare the properties of existing metrics given the premises that (1) queries may have multiple intents; (2) the likelihood of each intent given a query is available; and (3) graded relevance assessments are available for each intent. We compare a wide range of traditional and diversified IR metrics after adding graded relevance assessments to the TREC 2009 Web track diversity task test collection which originally had binary relevance assessments. Our primary criterion is discriminative power, which represents the reliability of a metric in an experiment. Our results show that diversified IR experiments with a given number of topics can be as reliable as traditional IR experiments with the same number of topics, provided that the right metrics are used. Moreover, we compare the intuitiveness of diversified IR metrics by closely examining the actual ranked lists from TREC. We show that a family of metrics called D -measures have several advantages over other metrics such as α-nDCG and Intent-Aware metrics.

The Reusability of a Diversified Search Test Collection

“…This simple approach has several drawbacks: the IA metric as defined above does not fully range between 0-1; in general it does not necessarily encourage diversity relative to relevance [9,24]; and it has limited discriminative power, i.e. the ability to draw reliable conclusions from statistical tests in an experiment [24,25].…”

“…Pr (i|q) = 1/|{i}| for any i) and graded relevance assessments are not utilised, the NTCIR INTENT task utilises these types of information by leveraging the "D " evaluation framework of Sakai and Song [24]. More specifically, a diversity version of normalised discounted cumulative gain (nDCG) [13] called D-nDCG is computed, based on the global gain which consolidates perintent graded relevance assessments and intent probabilities for each document.…”

“…It has been shown, in several experimental settings using NT-CIREVAL, that D( )-nDCG outperforms ERR-IA in terms of discriminative power and/or concordance tests, which reflect how often diversity metrics agree with simpler and more "intuitive" metrics such as precision and intent recall [24,25,17]. Sanderson et al [26] have tried to study how diversity metrics agree with user preferences, but their study treated each intent (i.e.…”

On the role of novelty for search result diversification

“…Both ERR-IA and a-nDCG have been shown to reward rankings that achieve a balance of coverage and novelty (Clarke et al 2011). Moreover, a-nDCG has been shown to possess a discriminative power at least as high as that of the traditional nDCG (Sakai and Song 2011). Following the standard TREC setting, unless otherwise noted, both metrics are reported at rank cutoff 20 (Clarke et al 2010).…”

mNIR: Diversifying Search Results Based on a Mixture of Novelty, Intention and Relevance

“…As a result, their approach effectively determines when and how to diversify the results for an unseen query. Sakai et al [14,15] proposed an alternative way to evaluate diversified search results, given intent probabilities and per-intent graded relevance assessments.…”