The Combination and Evaluation of Query Performance Prediction Methods

Abstract

Abstract. In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query's performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method's quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.

Estimating query difficulty for news prediction retrieval

“…Existing approaches to predicting query performance can be categorized wrt. two aspects [7]: 1) time of predicting (pre/post-retrieval) and 2) objective of task (difficulty, query rank, effectiveness). Pre-retrieval predictors work independently from a specific retrieval model and result documents, and such predictors are preferred to post-retrieval based methods because they are based solely on query terms, collection statistics and possibly external knowledge, such as WordNet or Wikipedia.…”

“…To measure the relationship between two terms, pointwise mutual information (PMI) is computed as suggested in [7]. PMI measures the term relationship by observing co-occurrence statistics of terms in a document collection.…”

“…PMI measures the term relationship by observing co-occurrence statistics of terms in a document collection. Two PMI-based predictors are proposed in [7] including the averaged PMI value and the maximum PMI value of all query term pairs. More detailed descriptions of different approaches to query difficulty estimation can be found in the book by Carmel and Yom-Tov [4], and references therein.…”

A Belief Model of Query Difficulty That Uses Subjective Logic

“…The main components of query difficulty have been defined as the textual expression of the query, the set of documents relevant to the query and the entire collection of documents, with experiments showing that query difficulty strongly depends on the distances between these components [4]. Finally, a recent overview of query difficulty with respect to performance prediction can be found in [9].…”

A Performance Prediction Approach to Enhance Collaborative Filtering Performance

“…According to whether or not the retrieval results are used in the prediction, the methods can be classified into pre-and post-retrieval approaches [10]. The first type has the advantage that the prediction can be taken into account to improve the retrieval process itself.…”