Using graded relevance assessments in IR evaluation

Kekäläinen, Jaana; Järvelin, Kalervo

doi:10.1002/asi.10137

Cited by 174 publications

(118 citation statements)

References 30 publications

Supporting

Mentioning

112

Contrasting

Unclassified

Order By: Relevance

“…We were only interested in data sets with more than two relevance levels, firstly because the calibration for binary relevance labels does not make too much sense, and secondly, because the difference between various learning algorithms can be more pronounced in the multi-label case. Note also that the general consensus in the IR community is that graded relevance labels are superior to the binary setup when large document collections are involved Kekäläinen and Järvelin 2002;Sakai 2007). The features were normalized querywise in the LETOR data sets (OHSUMED, MQ 2007 , MQ 2008 ) so we did not preprocess them.…”

Section: Data Setsmentioning

confidence: 99%

Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

et al. 2013

View full text Add to dashboard Cite

In subset ranking, the goal is to learn a ranking function that approximates a gold standard partial ordering of a set of objects (in our case, a set of documents retrieved for the same query). The partial ordering is given by relevance labels representing the relevance of documents with respect to the query on an absolute scale. Our approach consists of three simple steps. First, we train standard multi-class classifiers (AdaBoost.MH and multi-class SVM) to discriminate between the relevance labels. Second, the posteriors of multi-class classifiers are calibrated using probabilistic and regression losses in order to estimate the Bayes-scoring function which optimizes the Normalized Discounted Cumulative Gain (NDCG). In the third step, instead of selecting the best multi-class hyperparameters and the best calibration, we mix all the learned models in a simple ensemble scheme. Mach Learn (2013) 93:261-292 Our extensive experimental study is itself a substantial contribution. We compare most of the existing learning-to-rank techniques on all of the available large-scale benchmark data sets using a standardized implementation of the NDCG score. We show that our approach is competitive with conceptually more complex listwise and pairwise methods, and clearly outperforms them as the data size grows. As a technical contribution, we clarify some of the confusing results related to the ambiguities of the evaluation tools, and propose guidelines for future studies.

show abstract

Section: Data Setsmentioning

confidence: 99%

Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

et al. 2013

View full text Add to dashboard Cite

show abstract

“…This assumes binary relevance. Graded relevance assessment measures such as the Discounted Cumulative Gain (DGC) measure can also be tackled (Kekäläinen and Järvelin, 2002).…”

Section: Areas Of Leaning In Ir Supportable By Mcq'smentioning

confidence: 99%

Using Multiple Choice Questions to Assist Learning for Information Retrieval

MacFarlane

2011

The Information Retrieval Series

View full text Add to dashboard Cite

This is the accepted version of the paper.This version of the publication may differ from the final published version. Abstract A key issue in teaching and learning in information retrievalparticularly for library and information science students -is the gap in prior knowledge compared with the need for mathematics to conduct and evaluate searches. In this chapter, we examine the use of online Multiple Choice Questions to support these type of students, and narrow this gap between experience and knowledge. We provide some background in terms of related work and the use of MCQ's for assessment. The key areas of search which can be supported by this form of assessment are defined, and these are used to outline a proposed strategy for defining a series of questions to support learning. Permanent

show abstract

“…They can be negative or positive. Typically the relevance judgements are binary (0 or 1), but there is increasing interest in using graded relevance (Kekäläinen and Järvelin, 2002). The distribution of relevant documents differs significantly between collections (Hawking and Robertson, 2003), a further aspect any practitioner should think about when choosing a test collection.…”

Section: Benchmarking and Test Collectionsmentioning

confidence: 99%

Local search: A guide for the information retrieval practitioner

MacFarlane

Tuson

2009

Information Processing & Management

View full text Add to dashboard Cite

This is the unspecified version of the paper.This version of the publication may differ from the final published version. Permanent repository link AbstractThere are a number of combinatorial optimisation problems in information retrieval in which the use of local search methods are worthwhile. The purpose of this paper is to show how local search can be used to solve some well known tasks in Information Retrieval (IR), how previous research in the field is piecemeal, bereft of a structure and methodologically flawed, and to suggest more rigorous ways of applying local search methods to solve IR problems. We provide a Query based taxonomy for analysing the use of local search in IR tasks and an overview of issues such as fitness functions, statistical significance and test collections when conducting experiments on combinatorial optimisation problems. The paper gives a guide on the pitfalls and problems for IR practitioners who wish to use local search to solve their research issues, and gives practical advice on the use of such methods. The Query based taxonomy is a novel structure which can be used by the IR practitioner in order to examine the use of local search in IR.

show abstract

Using graded relevance assessments in IR evaluation

Cited by 174 publications

References 30 publications

Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

Using Multiple Choice Questions to Assist Learning for Information Retrieval

Local search: A guide for the information retrieval practitioner

Contact Info

Product

Resources

About