Query Performance Prediction Using Reference Lists

Shtok, Anna; Kurland, Oren; Carmel, David

doi:10.1145/2926790

Cited by 41 publications

(15 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, feedback could be further weaved into CLF by devising and integrating weighting schemes that account for this. We also plan to investigate the use of CLF as a method for query performance prediction (e.g., as a postretrieval predictor using reference lists [50], or as a candidate selection function in query transformation chain frameworks [48]). In terms of extending CLF, the weighting schemes themselves can be weighted (i.e., one weighting scheme may have more importance over others); e.g., using the linear combination fusion method [53] which assigns weights to each ranker being fused.…”

Section: Discussionmentioning

confidence: 99%

You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews

Scells

Zuccon

Koopman

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Coordination level matching is a ranking method originally proposed to rank documents given Boolean queries that is now several decades old. Rank fusion is a relatively recent method for combining runs from multiple systems into a single ranking, and has been shown to significantly improve the ranking. This paper presents a novel extension to coordination level matching, by applying rank fusion to each sub-clause of a Boolean query. We show that, for the tasks of systematic review screening prioritisation and stopping estimation, our method significantly outperforms the state-of-the-art learning to rank and bag-ofwords-based systems for this domain. Our fully automatic, unsupervised method has (i) the potential for significant real-world cost savings (ii) does not rely on any intervention from the user, and (iii) is significantly better at ranking documents given only a Boolean query in the context of systematic reviews when compared to other approaches.

show abstract

Section: Discussionmentioning

confidence: 99%

You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews

Scells

Zuccon

Koopman

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Results are likely to be less collection-biased; (3) Since the query set has a strong significant impact on the system effectiveness, zooming in the query sets is worth exploring. To do so, we consider the level of query difficulty since this is an active research direction [6,19,22]. Zooming in and considering individual query or groups of queries could help in understanding what are the main system failures and how to avoid them.…”

Section: Data Analysis Objectives and Methodsmentioning

confidence: 99%

Studying the Variability of System Setting Effectiveness by Data Analytics and Visualization

Déjean

Mothe²,

Ullah

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Search engines differ from their modules and parameters; defining the optimal system setting is challenging the more because of the complexity of a retrieval stream. The main goal of this study is to determine which are the most important system components and parameters in system setting, thus which ones should be tuned as the first priority. We carry out an extensive analysis of 20, 000 different system settings applied to three TREC ad-hoc collections. Our analysis includes zooming in and out the data using various data analysis methods such as ANOVA, CART, and data visualization. We found that the query expansion model is the most significant component that changes the system effectiveness, consistently across collections. Zooming in the queries, we show that the most significant component changes to the retrieval model when considering easy queries only. The results of our study are directly re-usable for the system designers and for system tuning.

show abstract

“…The first involves generating pre-retrieval prediction methods that rely on only the input query and features of the underlying document corpus [3,110,111,238]. The second line of inquiry allows the retrieved results list to be used for prediction, resulting in a richer feature space and enhanced methods of QPP [47,73,112,214,226,271]. The third and final broad line of work focuses on the actual prediction framework itself, with various formalizations of the problem being explored, including probabilistic frameworks [140], distance-based measures [47], and neural approaches [266].…”

Section: Query Performance Predictionmentioning

confidence: 99%

Managing tail latency in large scale information retrieval systems

Mackenzie

2020

SIGIR Forum

View full text Add to dashboard Cite

As both the availability of internet access and the prominence of smart devices continue to increase, data is being generated at a rate faster than ever before. This massive increase in data production comes with many challenges, including efficiency concerns for the storage and retrieval of such large-scale data. However, users have grown to expect the sub-second response times that are common in most modern search engines, creating a problem --- how can such large amounts of data continue to be served efficiently enough to satisfy end users? This dissertation investigates several issues regarding tail latency in large-scale information retrieval systems. Tail latency corresponds to the high percentile latency that is observed from a system --- in the case of search, this latency typically corresponds to how long it takes for a query to be processed. In particular, keeping tail latency as low as possible translates to a good experience for all users, as tail latency is directly related to the worst-case latency and hence, the worst possible user experience. The key idea in targeting tail latency is to move from questions such as "what is the median latency of our search engine?" to questions which more accurately capture user experience such as "how many queries take more than 200 ms to return answers?" or "what is the worst case latency that a user may be subject to, and how often might it occur?" While various strategies exist for efficiently processing queries over large textual corpora, prior research has focused almost entirely on improvements to the average processing time or cost of search systems. As a first contribution, we examine some state-of-the-art retrieval algorithms for two popular index organizations, and discuss the trade-offs between them, paying special attention to the notion of tail latency. This research uncovers a number of observations that are subsequently leveraged for improved search efficiency and effectiveness. We then propose and solve a new problem, which involves processing a number of related query variations together, known as multi-queries , to yield higher quality search results. We experiment with a number of algorithmic approaches to efficiently process these multi-queries, and report on the cost, efficiency, and effectiveness trade-offs present with each. Finally, we examine how predictive models can be used to improve the tail latency and end-to-end cost of a commonly used multi-stage retrieval architecture without impacting result effectiveness. By combining ideas from numerous areas of information retrieval, we propose a prediction framework which can be used for training and evaluating several efficiency/effectiveness trade-off parameters, resulting in improved trade-offs between cost, result quality, and tail latency.

show abstract

Query Performance Prediction Using Reference Lists

Cited by 41 publications

References 46 publications

You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews

You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews

Studying the Variability of System Setting Effectiveness by Data Analytics and Visualization

Managing tail latency in large scale information retrieval systems

Contact Info

Product

Resources

About