Standard Deviation as a Query Hardness Estimator

Improved query performance prediction using standard deviation

“…Query performance prediction (QPP) has been a vibrant area of IR research over the last decade [2,4,3]. The motivation for QPP is that, if we can predict the performance of a query for a given system, we can automatically develop different strategies for dealing with these different queries.…”

Navigating the User Query Space

“…Fundamentally, retrieval predictors can be divided into two classes: pre-retrieval [7,6,12] and post-retrieval [3,10,11] predictors. Pre-retrieval predictors use features from the query, document and collection before a query has been processed in order to ascertain its performance.…”

“…One of the earliest approaches to QPP has been that of the clarity score [3], which measures the KL-divergence between the query and collection model in a language modelling framework. Recent research has shown that the standard deviation (σ) of scores in a ranked list is a good predictor of query performance [10,11] for the traditional QPP task. It has also been shown [10] that even better prediction can be obtained if a variable cut-off point is used (i.e.…”

“…Recent research has shown that the standard deviation (σ) of scores in a ranked list is a good predictor of query performance [10,11] for the traditional QPP task. It has also been shown [10] that even better prediction can be obtained if a variable cut-off point is used (i.e. different cut-off points for different queries).…”

“…Some of the highest performing post-retrieval predictors from the literature are query clarity (clarity), standard deviation at 100 documents (σ 100 ), a normalised version of standard deviation at 100 documents (i.e. the ncq predictor [11]), the maximum standard deviation in the ranked-list (σ max ) [10]. We also use two new predictors that calculate the standard deviation using a variable cut-off point (σ 50% ), and a query length normalised version of that predictor (n(σ 50% )) [5].…”