An Effectiveness Measure for Ambiguous and Underspecified Queries

Clarke, Charles L. A.; Kolla, Maheedhar; Vechtomova, Olga

doi:10.1007/978-3-642-04417-5_17

Cited by 99 publications

(59 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(At TREC, precision and intent recall have been used for similar purposes. ) Clarke, Kolla and Vechtomova [12] proposed to utilise the ambiguous/faceted topic tags for diversity evaluation, but this has not been put into practice. Sakai [17] proposed to utilise the informational/navigational intent tags, to encourage systems to allocate more space to informational ones in the search engine result page.…”

Section: Diversity Evaluation Metricsmentioning

confidence: 99%

The Reusability of a Diversified Search Test Collection

Sakai

Dou

Song

et al. 2012

Information Retrieval Technology

View full text Add to dashboard Cite

Traditional "ad hoc" test collections, typically built based on depth-100 pools, are often used a posteriori by non-contributors, i.e., research groups that did not contribute the pools. The Leave One Out (LOO) test is useful for testing whether the test collections are actually reusable: that is, whether the non-contributors can be evaluated fairly relative to the contributors' official performances. In contrast, at the recent web search result diversification tasks of TREC and NTCIR, diversity test collections have been built using shallow pools: the pool depths lie between 20 and 40. Thus it is unlikely that these diversity test collections are reusable: in fact, the organisers of these diversity tasks never claimed that they are. Nevertheless, these collections are also used a posteriori by non-contributors. In light of this, Sakai et al. [21] demonstrated by means of LOO tests that the NTCIR-9 INTENT-1 Chinese diversity test collection is not reusable, and also showed that condensed-list evaluation metrics generally provide better estimates of the noncontributors' true performances than raw evaluation metrics. This paper generalises and strengthens their findings through LOO tests with the latest TREC 2012 diversity test collection.

show abstract

Section: Diversity Evaluation Metricsmentioning

confidence: 99%

The Reusability of a Diversified Search Test Collection

Sakai

Dou

Song

et al. 2012

Information Retrieval Technology

View full text Add to dashboard Cite

show abstract

“…Thus, an IR system aims to maximise the user's satisfaction by retrieving documents which cover all the relevant subtopics in the ranking. Given a test collection, where the subtopics of the relevant documents have been identified, the effectiveness of the retrieval system can be measured in several ways [4,6,7,11]. The three main measures employed are S-recall, S-Mean Reciprocal Rank and S-precision.…”

Section: Background and Related Workmentioning

confidence: 99%

“…When results need to be relevant but also diverse, as is the case in subtopic retrieval, the independence assumption made by the PRP is unrealistic. This is because the PRP neglects relationships between documents at relevance level ignoring the fact that a previous document may already contain similar relevant material [4,5,6,7].…”

Section: Introductionmentioning

confidence: 99%

Using the Quantum Probability Ranking Principle to Rank Interdependent Documents

Zuccon

Azzopardi

2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A known limitation of the Probability Ranking Principle (PRP) is that it does not cater for dependence between documents. Recently, the Quantum Probability Ranking Principle (QPRP) has been proposed, which implicitly captures dependencies between documents through "quantum interference". This paper explores whether this new ranking principle leads to improved performance for subtopic retrieval, where novelty and diversity is required. In a thorough empirical investigation, models based on the PRP, as well as other recently proposed ranking strategies for subtopic retrieval (i.e. Maximal Marginal Relevance (MMR) and Portfolio Theory(PT)), are compared against the QPRP. On the given task, it is shown that the QPRP outperforms these other ranking strategies. And unlike MMR and PT, one of the main advantages of the QPRP is that no parameter estimation/tuning is required; making the QPRP both simple and effective. This research demonstrates that the application of quantum theory to problems within information retrieval can lead to significant improvements.

show abstract

“…Much prior work on this problem has focused on manually-tuned methods for generating diverse results [2][3][4][5][6]. Some learning approaches exist as well and have been shown to outperform manually tuned methods [7][8][9][10].…”

Section: Introductionmentioning

confidence: 99%

Learning Socially Optimal Information Systems from Egoistic Users

Raman

Joachims

2013

Advanced Information Systems Engineering

View full text Add to dashboard Cite

Abstract. Many information systems aim to present results that maximize the collective satisfaction of the user population. The product search of an online store, for example, needs to present an appropriately diverse set of products to best satisfy the different tastes and needs of its user population. To address this problem, we propose two algorithms that can exploit observable user actions (e.g. clicks) to learn how to compose diverse sets (and rankings) that optimize expected utility over a distribution of utility functions. A key challenge is that individual users evaluate and act according to their own utility function, but that the system aims to optimize collective satisfaction. We characterize the behavior of our algorithms by providing upper bounds on the social regret for a class of submodular utility functions in the coactive learning model. Furthermore, we empirically demonstrate the efficacy and robustness of the proposed algorithms for the problem of search result diversification.

show abstract

An Effectiveness Measure for Ambiguous and Underspecified Queries

Cited by 99 publications

References 8 publications

The Reusability of a Diversified Search Test Collection

The Reusability of a Diversified Search Test Collection

Using the Quantum Probability Ranking Principle to Rank Interdependent Documents

Learning Socially Optimal Information Systems from Egoistic Users

Contact Info

Product

Resources

About