Groupwise Query Performance Prediction with BERT

Chen, Xiaoyang; He, Ben; Sun, Le

doi:10.1007/978-3-030-99739-7_8

Cited by 9 publications

(4 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, BERT-QPP [2] encodes semantics via BERT, but directly fine-tunes it to predict query performance based on the first retrieved document. Subsequent approaches extend BERT-QPP by employing a groupwise predictor to jointly learn from multiple queries and documents [5] or by transforming its pointwise regression into a classification task [12]. Since we did not consider multiple formulations, we did not experiment with such approach in our empirical evaluation.…”

Section: Related Workmentioning

confidence: 99%

Query Performance Prediction for Neural IR: Are We There Yet?

Faggioli¹,

Formal²,

Marchesin³

et al. 2023

Preprint

View full text Add to dashboard Cite

Evaluation in Information Retrieval (IR) relies on post-hoc empirical procedures, which are time-consuming and expensive operations. To alleviate this, Query Performance Prediction (QPP) models have been developed to estimate the performance of a system without the need for human-made relevance judgements. Such models, usually relying on lexical features from queries and corpora, have been applied to traditional sparse IR methods -with various degrees of success. With the advent of neural IR and large Pre-trained Language Models, the retrieval paradigm has significantly shifted towards more semantic signals. In this work, we study and analyze to what extent current QPP models can predict the performance of such systems. Our experiments consider seven traditional bag-of-words and seven BERT-based IR approaches, as well as nineteen state-of-the-art QPPs evaluated on two collections, Deep Learning '19 and Robust '04. Our findings show that QPPs perform statistically significantly worse on neural IR systems. In settings where semantic signals are prominent (e.g., passage retrieval), their performance on neural models drops by as much as 10% compared to bag-of-words approaches. On top of that, in lexical-oriented scenarios, QPPs fail to predict performance for neural IR systems on those queries where they differ from traditional approaches the most.

show abstract

Section: Related Workmentioning

confidence: 99%

Query Performance Prediction for Neural IR: Are We There Yet?

Faggioli¹,

Formal²,

Marchesin³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…These two datasets have different properties which help to demonstrate the properties of our GERank model in different scenarios. We also use a widely used TREC (https://trec.nist.gov/, accessed on 10 August 2022) dataset, Robust04, which is a TREC collection with documents from the news domain that many previous and most recent works used for retrieval evaluation purposes [64][65][66][67]. In the following, we describe each of these datasets in more details:…”

Section: Datasetsmentioning

confidence: 99%

Learning to Co-Embed Queries and Documents

Lin

et al. 2022

Electronics

View full text Add to dashboard Cite

Learning to Rank (L2R) methods that utilize machine learning techniques to solve the ranking problems have been widely studied in the field of information retrieval. Existing methods usually concatenate query and document features as training input, without explicit understanding of relevance between queries and documents, especially in pairwise based ranking approach. Thus, it is an interesting question whether we can devise an algorithm that effectively describes the relation between queries and documents to learn a better ranking model without incurring huge parameter costs. In this paper, we present a Gaussian Embedding model for Ranking (GERank), an architecture for co-embedding queries and documents, such that each query or document is represented by a Gaussian distribution with mean and variance. Our GERank optimizes an energy-based loss based on the pairwise ranking framework. Additionally, the KL-divergence is utilized to measure the relevance between queries and documents. Experimental results on two LETOR datasets and one TREC dataset demonstrate that our model obtains a remarkable improvement in the ranking performance compared with the state-of-the-art retrieval models.

show abstract

“…Recently, supervised approaches have been shown to outperform their unsupervised counterparts [1,3,7,24]. The increase in effectiveness, however, comes at the cost of necessitating the availability of a set of queries with ground-truth relevance assessments, which are used for supervised training.…”

Section: Introductionmentioning

confidence: 99%

“…Motivated by the success of BERT-based QPP models [1], a groupwise query estimation framework is proposed [3] that utilizes both cross-query and cross-document information across groups to learn the query performance predictor. Similar to BERT-QPP, this is also a regression-based model that predicts individual score for each query-document pair.…”

Section: Introductionmentioning

confidence: 99%