PARADE: Passage Representation Aggregation for Document Reranking

Li, Canjia; Yates, Andrew; MacAvaney, Sean; He, Ben; Sun, Yingfei

doi:10.48550/arxiv.2008.09093

Cited by 25 publications

(50 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To test the effectiveness of our proposed dense PRF approach, we compare with four families of baseline models, for which we vary the use of a BERT-based reranker (namely BERT or ColBERT). For the BERT reranker, we use OpenNIR [21] and capreolus/ bert-base-msmarco fine-tuned model from [19]. For the ColBERT reranker, unless otherwise noted, we use the existing pre-indexed ColBERT representation of documents for efficient reranking.…”

Section: Baselinesmentioning

confidence: 99%

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Wang

Macdonald

Tonellotto

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Pseudo-relevance feedback mechanisms, from Rocchio to the relevance models, have shown the usefulness of expanding and reweighting the users' initial queries using information occurring in an initial set of retrieved documents, known as the pseudo-relevant set. Recently, dense retrieval -through the use of neural contextual language models such as BERT for analysing the documents' and queries' contents and computing their relevance scores -has shown a promising performance on several information retrieval tasks still relying on the traditional inverted index for identifying documents relevant to a query. Two different dense retrieval families have emerged: the use of single embedded representations for each passage and query (e.g. using BERT's [CLS] token), or via multiple representations (e.g. using an embedding for each token of the query and document). In this work, we conduct the first study into the potential for multiple representation dense retrieval to be enhanced using pseudo-relevance feedback. In particular, based on the pseudo-relevant set of documents identified using a first-pass dense retrieval, we extract representative feedback embeddings (using KMeans clustering) -while ensuring that these embeddings discriminate among passages (based on IDF) -which are then added to the query representation. These additional feedback embeddings are shown to both enhance the effectiveness of a reranking as well as an additional dense retrieval operation. Indeed, experiments on the MSMARCO passage ranking dataset show that MAP can be improved by upto 26% on the TREC 2019 query set and 10% on the TREC 2020 query set by the application of our proposed ColBERT-PRF method on a ColBERT dense retrieval approach.

show abstract

Section: Baselinesmentioning

confidence: 99%

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Wang

Macdonald

Tonellotto

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

show abstract

“…PARADE model divides a document into a number of segments and encodes each of them using Transformer. Encoded segments are again fed to another Transformer to get the final document level score [9] * . QDS-Transformer encodes the long texts with fixed patterns of attentions which allow local attention among neighboring content tokens, and combines them with long distance attention [5].…”

Section: Arxiv:210904611v1 [Csir] 10 Sep 2021 2 Related Workmentioning

confidence: 99%

“…The first addresses the problem by dividing the document tokens into segments of similar length and applying the self-attention mechanism only within those segments (local self-attention) then aggregating vectors from segments to get the final score. [1,5,9,10]. A second approach also starts by dividing the document tokens into segments and treating each of them as if it were a document in its own right.…”

Section: Introductionmentioning

confidence: 99%

Query-driven Segment Selection for Ranking Long Documents

Kim

Rahimi

Bonab

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Transformer-based rankers have shown state-of-the-art performance. However, their self-attention operation is mostly unable to process long sequences. One of the common approaches to train these rankers is to heuristically select some segments of each document, such as the first segment, as training data. However, these segments may not contain the query-related parts of documents.To address this problem, we propose query-driven segment selection from long documents to build training data. The segment selector provides relevant samples with more accurate labels and non-relevant samples which are harder to be predicted. The experimental results show that the basic BERT-based ranker trained with the proposed segment selector significantly outperforms that trained by the heuristically selected segments, and performs equally to the state-of-the-art model with localized self-attention that can process longer input sequences. Our findings open up new direction to design efficient transformer-based rankers. CCS CONCEPTS• Information systems → Retrieval models and ranking.

show abstract

“…We use the keyword version of queries, corresponding to the title fields of TREC topics [14,31]. We experimented with vanilla BERT [15] as the neural ranking model, as it is the core of recent state-of-the-art IR methods [14,29,26]. To the best of our knowledge, most text-based IR neural models are trained with a pointwise or pairwise loss [26,29].…”

Section: Experiments On Text-based Ir (Rq4)mentioning

confidence: 99%

“…We experimented with vanilla BERT [15] as the neural ranking model, as it is the core of recent state-of-the-art IR methods [14,29,26]. To the best of our knowledge, most text-based IR neural models are trained with a pointwise or pairwise loss [26,29]. A challenge in this experiment was then to use a listwise loss on a BERT model.…”

Section: Experiments On Text-based Ir (Rq4)mentioning

confidence: 99%

SmoothI: Smooth Rank Indicators for Differentiable IR Metrics

Thonet,

Cinar,

Gaussier

et al. 2021

Preprint

View full text Add to dashboard Cite

Information retrieval (IR) systems traditionally aim to maximize metrics built on rankings, such as precision or NDCG. However, the non-differentiability of the ranking operation prevents direct optimization of such metrics in state-of-the-art neural IR models, which rely entirely on the ability to compute meaningful gradients. To address this shortcoming, we propose SmoothI, a smooth approximation of rank indicators that serves as a basic building block to devise differentiable approximations of IR metrics. We further provide theoretical guarantees on SmoothI and derived approximations, showing in particular that the approximation errors decrease exponentially with an inverse temperature-like hyperparameter that controls the quality of the approximations. Extensive experiments conducted on four standard learning-to-rank datasets validate the efficacy of the listwise losses based on SmoothI, in comparison to previously proposed ones. Additional experiments with a vanilla BERT ranking model on a text-based IR task also confirm the benefits of our listwise approach.

show abstract

PARADE: Passage Representation Aggregation for Document Reranking

Cited by 25 publications

References 25 publications

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Query-driven Segment Selection for Ranking Long Documents

SmoothI: Smooth Rank Indicators for Differentiable IR Metrics

Contact Info

Product

Resources

About