Predicting Query Performance by Query-Drift Estimation

Abstract

Predicting query performance, that is, the effectiveness of a search performed in response to a query, is a highly important and challenging problem. Our novel approach to addressing this challenge is based on estimating the potential amount of query drift in the result list, i.e., the presence (and dominance) of aspects or topics not related to the query in top-retrieved documents. We argue that query-drift can potentially be estimated by measuring the diversity (e.g., standard deviation) of the retrieval scores of these documents. Empirical evaluation demonstrates the prediction effectiveness of our approach for several retrieval models. Specifically, the prediction success is better, over most tested TREC corpora, than that of state-of-the-art prediction methods.

Navigating the User Query Space

“…Fundamentally, retrieval predictors can be divided into two classes: pre-retrieval [7,6,12] and post-retrieval [3,10,11] predictors. Pre-retrieval predictors use features from the query, document and collection before a query has been processed in order to ascertain its performance.…”

“…One of the earliest approaches to QPP has been that of the clarity score [3], which measures the KL-divergence between the query and collection model in a language modelling framework. Recent research has shown that the standard deviation (σ) of scores in a ranked list is a good predictor of query performance [10,11] for the traditional QPP task. It has also been shown [10] that even better prediction can be obtained if a variable cut-off point is used (i.e.…”

Predicting the effectiveness of queries and retrieval systems

“…Shtok et al [130] and Perez-Iglesias and Araujo [118] experiment with estimating the coverage of query aspects in the ranked list of results by deriving the retrieval scores' standard deviation, possibly normalized by a query dependent corpus statistic. It is hypothesized, that a high standard deviation indicates a high "query-commitment" [130] and the absence of aspects unrelated to the query.…”

“…It is hypothesized, that a high standard deviation indicates a high "query-commitment" [130] and the absence of aspects unrelated to the query. This indicates a result list of high quality.…”

“…Our results are based on the top 100 documents as recommended in [130]. Similarly to the Robustness approach, small changes in the parameter do not affect the performance of this approach.…”