A Unified Framework for Learned Sparse Retrieval

Nguyen, T. Q.; MacAvaney, Sean; Yates, Andrew

doi:10.1007/978-3-031-28241-6_7

Cited by 14 publications

(3 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conduct a similar Pareto analysis for zero-shot retrieval for our work. LSR [31] is a recent concurrent work with ours. The authors provide a toolkit focused on sparse-retrieval model training and evaluate different training settings with in-domain datasets such as MS MARCO [32].…”

Section: Related Workmentioning

confidence: 90%

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

Nandan

Wang

Gurevych

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Traditionally, sparse retrieval systems relied on lexical representations to retrieve documents, such as BM25, dominated information retrieval tasks. With the onset of pre-trained transformer models such as BERT, neural sparse retrieval has led to a new paradigm within retrieval. Despite the success, there has been limited software supporting different sparse retrievers running in a unified, common environment. This hinders practitioners from fairly comparing different sparse models and obtaining realistic evaluation results. Another missing piece is, that a majority of prior work evaluates sparse retrieval models on in-domain retrieval, i.e. on a single dataset: MS MARCO. However, a key requirement in practical retrieval systems requires models that can generalize well to unseen out-of-domain, i.e. zero-shot retrieval tasks. In this work, we provide SPRINT, a unified python toolkit based on Pyserini and Lucene, supporting a common interface for evaluating neural sparse retrieval. The toolkit currently includes five built-in models: uni-COIL, DeepImpact, SPARTA, TILDEv2 and SPLADEv2. Users can also easily add customized models by defining their term weighting method. Using our toolkit, we establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. Our results demonstrate that SPLADEv2 achieves the best average score of 0.470 nDCG@10 on BEIR amongst all neural sparse retrievers. In this work, we further uncover the reasons behind its performance gain. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document which is often crucial for its performance gains, i.e. a limitation among its other sparse counterparts. We provide our SPRINT toolkit, models, and data used in our experiments publicly here: https://github.com/thakur-nandan/sprint.

show abstract

Section: Related Workmentioning

confidence: 90%

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

Nandan

Wang

Gurevych

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…PROP proposed a novel representative words prediction training task [63], while B-PROP further improves upon PROP by replacing PROP's classical unigram language model with a more powerful BERTbased contextual language model [64]. Other researchers trade off PLM effectiveness for efficiency by utilizing the PLM to improve document indexing [19,77], pre-computing intermediate Transformer representations [27,42,47,65], selecting query-aware key blocks within a document for input squeezing [48,55], using the PLM to build sparse representations [25,56,66,68,73,112,114], weighting offline pseudo-query and document relevance [11], or reducing the number of Transformer layers [34,36,72].…”

Section: Related Workmentioning

confidence: 99%

PARADE: Passage Representation Aggregation forDocument Reranking

Yates

MacAvaney

et al. 2023

ACM Trans. Inf. Syst.

Self Cite

View full text Add to dashboard Cite

Pre-trained transformer models, such as BERT and T5, have shown to be highly effective at ad-hoc passage and document ranking. Due to the inherent sequence length limits of these models, they need to process document passages one at a time rather than processing the entire document sequence at once. Although several approaches for aggregating passage-level signals into a document-level relevance score have been proposed, there has yet to be an extensive comparison of these techniques. In this work, we explore strategies for aggregating relevance signals from a document’s passages into a final ranking score. We find that passage representation aggregation techniques can significantly improve over score aggregation techniques proposed in prior work, such as taking the maximum passage score. We call this new approach PARADE. In particular, PARADE can significantly improve results on collections with broad information needs where relevance signals can be spread throughout the document (such as TREC Robust04 and GOV2). Meanwhile, less complex aggregation techniques may work better on collections with an information need that can often be pinpointed to a single passage (such as TREC DL and TREC Genomics). We also conduct efficiency analyses and highlight several strategies for improving transformer-based aggregation.

show abstract

“…However, the evolution of information retrieval has integrated machine learning algorithms to generate document vectors containing term scores learned from the documents, akin to traditional term frequency. This integration of machine learning, primarily based on neural networks, has led to the emergence of Neural Information Retrieval [6].…”

Section: Introductionmentioning

confidence: 99%

On Embedding Implementations in Text Ranking and Classification Employing Graphs

Kalogeropoulos,

Ioannou,

Stathopoulos

et al. 2024

Electronics

View full text Add to dashboard Cite

This paper aims to enhance the Graphical Set-based model (GSB) for ranking and classification tasks by incorporating node and word embeddings. The model integrates a textual graph representation with a set-based model for information retrieval. Initially, each document in a collection is transformed into a graph representation. The proposed enhancement involves augmenting the edges of these graphs with embeddings, which can be pretrained or generated using Word2Vec and GloVe models. Additionally, an alternative aspect of our proposed model consists of the Node2Vec embedding technique, which is applied to a graph created at the collection level through the extension of the set-based model, providing edges based on the graph’s structural information. Core decomposition is utilized as a method for pruning the graph. As a byproduct of our information retrieval model, we explore text classification techniques based on our approach. Node2Vec embeddings are generated by our graphs and are applied in order to represent the different documents in our collections that have undergone various preprocessing methods. We compare the graph-based embeddings with the Doc2Vec and Word2Vec representations to elaborate on whether our approach can be implemented on topic classification problems. For that reason, we then train popular classifiers on the document embeddings obtained from each model.

show abstract

A Unified Framework for Learned Sparse Retrieval

Cited by 14 publications

References 45 publications

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

PARADE: Passage Representation Aggregation forDocument Reranking

On Embedding Implementations in Text Ranking and Classification Employing Graphs

Contact Info

Product

Resources

About