Complement Lexical Retrieval Model with Semantic Residual Embeddings

Gao, Luyu; Dai, Zhuyun; Chen, Tongfei; Fan, Zhen; Durme, Benjamin Van; Callan, Jamie

doi:10.1007/978-3-030-72113-8_10

Cited by 82 publications

(70 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We implement approximate search to retrieve using a linear combination of two systems by re-ranking n-best top scoring candidates from each system. Prior and concurrent work has also used hybrid sparse-dense models (Guo et al, 2016a;Seo et al, 2019;Karpukhin et al, 2020;Ma et al, 2020;Gao et al, 2020). Our contribution is to assess the impact of sparse-dense hybrids as the document length grows.…”

Section: Sparse-dense Hybrids (Hybrid)mentioning

confidence: 99%

Sparse, Dense, and Attentional Representations for Text Retrieval

Luan

Eisenstein

Toutanova

et al. 2021

Transactions of the Association for Computational Linguistics

189

View full text Add to dashboard Cite

Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.

show abstract

Section: Sparse-dense Hybrids (Hybrid)mentioning

confidence: 99%

Sparse, Dense, and Attentional Representations for Text Retrieval

Luan

Eisenstein

Toutanova

et al. 2021

Transactions of the Association for Computational Linguistics

189

View full text Add to dashboard Cite

show abstract

“…In the context of transformers, the general setup of ranking with dense representations involves learning transformer-based encoders that convert queries and texts into dense, fixed-size vectors. In the simplest approach, ranking becomes the problem of approximate nearest neighbor (ANN) search based on some simple metric such as cosine similarity Xiong et al, 2020;Lu et al, 2020;Reimers and Gurevych, 2019;Gao et al, 2020b;Karpukhin et al, 2020;Qu et al, 2020;Hofstätter et al, 2020a;Lin et al, 2020b). However, recognizing that accurate ranking cannot be captured via simple metrics, researchers have explored using more complex machinery to compare dense representations (Humeau et al, 2020;Khattab and Zaharia, 2020).…”

Section: Learned Dense Representationsmentioning

confidence: 99%

Pretrained Transformers for Text Ranking: BERT and Beyond

Yates¹,

Nogueira²,

Lin³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

“…A common choice for dense retrieval is to finetune a transformer network like BERT (Devlin et al, 2018) on a given training corpus with queries and relevant documents (Guo et al, 2020;Guu et al, 2020;Gao et al, 2020;Karpukhin et al, 2020;Luan et al, 2020). Recent work showed that combining dense approaches with sparse, lexical approaches can further boost the performance (Luan et al, 2020;Gao et al, 2020). While the approaches have been tested on various information and question answering retrieval datasets, the performance was only evaluated on fixed, rather small indexes.…”

Section: Related Workmentioning

confidence: 99%

“…by using cosine-similarity. Out-performance over sparse lexical approaches has been shown for various datasets (Gillick et al, 2018;Guo et al, 2020;Guu et al, 2020;Gao et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes

Reimers¹,

Gurevych²

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Information Retrieval using dense lowdimensional representations recently became popular and showed out-performance to traditional sparse-representations like BM25. However, no previous work investigated how dense representations perform with large index sizes. We show theoretically and empirically that the performance for dense representations decreases quicker than sparse representations for increasing index sizes. In extreme cases, this can even lead to a tipping point where at a certain index size sparse representations outperform dense representations. We show that this behavior is tightly connected to the number of dimensions of the representations: The lower the dimension, the higher the chance for false positives, i.e. returning irrelevant documents.

show abstract

Complement Lexical Retrieval Model with Semantic Residual Embeddings

Cited by 82 publications

References 22 publications

Sparse, Dense, and Attentional Representations for Text Retrieval

Sparse, Dense, and Attentional Representations for Text Retrieval

Pretrained Transformers for Text Ranking: BERT and Beyond

The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes

Contact Info

Product

Resources

About