A statistical method for system evaluation using incomplete judgments

Aslam, Javed A.; Pavlu, Virgil; Yılmaz, Emine

doi:10.1145/1148170.1148263

Cited by 124 publications

(100 citation statements)

References 7 publications

Supporting

Mentioning

100

Contrasting

Order By: Relevance

“…This ensures a sufficiently large training set for the initial classifier without losing much in the query selection performance. 2 The results on P @100 are not reported due to lack of space. As seen in Table 2, the required subset sizes for τ ={0.7, 0.8, 0.9} are statistically significantly smaller than those required for random sampling.…”

Section: Results Of the Web Datamentioning

confidence: 99%

“…However, other sources of uncertainty could be considered. Recent research is particularly concerned with measuring uncertainty of the systems performance due to (i ) partial relevance judgments [2,4,20] and (ii ) errors in the relevance judgments made by human assessors [6,12]. Our future work will expand the theoretical model to incorporate additional sources of uncertainty and explore more general cost models for constructing test collections.…”

Section: Discussionmentioning

confidence: 99%

“…In order to make relevance assessments feasible, various document selection methods are suggested [2,4]. These methods reduce the number of documents that need to be judged per query and are often used in conjunction with evaluation metrics [2,4,20] designed for shallow sets of relevance judgments.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

An uncertainty-aware query selection model for evaluation of IR systems

Hosseini

Cox

Milić-Frayling

et al. 2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

We propose a mathematical framework for query selection as a mechanism for reducing the cost of constructing information retrieval test collections. In particular, our mathematical formulation explicitly models the uncertainty in the retrieval effectiveness metrics that is introduced by the absence of relevance judgments. Since the optimization problem is computationally intractable, we devise an adaptive query selection algorithm, referred to as Adaptive, that provides an approximate solution. Adaptive selects queries iteratively and assumes that no relevance judgments are available for the query under consideration. Once a query is selected, the associated relevance assessments are acquired and then used to aid the selection of subsequent queries. We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries. Our experimental results show that the queries chosen by Adaptive produce reliable performance ranking of systems. The ranking is better correlated with the actual systems ranking than the rankings produced by queries that were selected using the considered baseline methods.

show abstract

Section: Results Of the Web Datamentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

An uncertainty-aware query selection model for evaluation of IR systems

Hosseini

Cox

Milić-Frayling

et al. 2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…Besides sampling queries, it is also possible to sample subsets of documents to be labeled for a given query. Carterette et al [18] use document sampling to decide which of two ranking functions achieves higher precision at k. Aslam et al [19] use document sampling to obtain unbiased estimates of mean average precision and mean R-precision. Carterette and Smucker [20] study statistical significance testing from reduced document sets.…”

Section: Related Workmentioning

confidence: 99%

Active evaluation of ranking functions based on graded relevance

et al. 2013

View full text Add to dashboard Cite

Abstract. Evaluating the quality of ranking functions is a core task in web search and other information retrieval domains. Because query distributions and item relevance change over time, ranking models often cannot be evaluated accurately on held-out training data. Instead, considerable effort is spent on manually labeling the relevance of query results for test queries in order to track ranking performance. We address the problem of estimating ranking performance as accurately as possible on a fixed labeling budget. Estimates are based on a set of most informative test queries selected by an active sampling distribution. Query labeling costs depend on the number of result items as well as item-specific attributes such as document length. We derive cost-optimal sampling distributions for the commonly used performance measures Discounted Cumulative Gain (DCG) and Expected Reciprocal Rank (ERR). Experiments on web search engine data illustrate significant reductions in labeling costs.

show abstract

“…With small numbers of relevance assessments (say 1 or 2) it is difficult to use BPREF measure as there are too few pairs to usefully distinguish between objects leading to severe overfitting. There is a question on the stability of the BPREF function at low sampling rates -correlations between systems rankings tend to deteriorate at this level (Aslam et al, 2006). However, it is not clear that using little evidence is much use for optimisation problems in IR (see section 6.1 above).…”

Section: Equationmentioning

confidence: 99%

Local search: A guide for the information retrieval practitioner

MacFarlane

Tuson

2009

Information Processing & Management

View full text Add to dashboard Cite

This is the unspecified version of the paper.This version of the publication may differ from the final published version. Permanent repository link AbstractThere are a number of combinatorial optimisation problems in information retrieval in which the use of local search methods are worthwhile. The purpose of this paper is to show how local search can be used to solve some well known tasks in Information Retrieval (IR), how previous research in the field is piecemeal, bereft of a structure and methodologically flawed, and to suggest more rigorous ways of applying local search methods to solve IR problems. We provide a Query based taxonomy for analysing the use of local search in IR tasks and an overview of issues such as fitness functions, statistical significance and test collections when conducting experiments on combinatorial optimisation problems. The paper gives a guide on the pitfalls and problems for IR practitioners who wish to use local search to solve their research issues, and gives practical advice on the use of such methods. The Query based taxonomy is a novel structure which can be used by the IR practitioner in order to examine the use of local search in IR.

show abstract

A statistical method for system evaluation using incomplete judgments

Cited by 124 publications

References 7 publications

An uncertainty-aware query selection model for evaluation of IR systems

An uncertainty-aware query selection model for evaluation of IR systems

Active evaluation of ranking functions based on graded relevance

Local search: A guide for the information retrieval practitioner

Contact Info

Product

Resources

About