CEQE: Contextualized Embeddings for Query Expansion

Abstract

In this work we leverage recent advances in context-sensitive language models to improve the task of query expansion. Contextualized word representation models, such as ELMo and BERT, are rapidly replacing static embedding models. We propose a new model, Contextualized Embeddings for Query Expansion (CEQE), that utilizes queryfocused contextualized embedding vectors. We study the behavior of contextual representations generated for query expansion in ad-hoc document retrieval. We conduct our experiments on probabilistic retrieval models as well as in combination with neural ranking models. We evaluate CEQE on two standard TREC collections: Robust and Deep Learning. We find that CEQE outperforms static embedding-based expansion methods on multiple collections (by up to 18% on Robust and 31% on Deep Learning on average precision) and also improves over proven probabilistic pseudo-relevance feedback (PRF) models. We further find that multiple passes of expansion and reranking result in continued gains in effectiveness with CEQE-based approaches outperforming other approaches. The final model incorporating neural and CEQE-based expansion score achieves gains of up to 5% in P@20 and 2% in AP on Robust over the state-of-the-art transformer-based re-ranking model, Birch.

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

“…Other approaches such as DeepCT [9] and doc2query [24,25] use neural models to augment documents before indexing using a traditional inverted index. CEQE [23] generates words to expand the initial query, which is then executed on the inverted index. However, returning the BERT embeddings back to textual word forms can result in polysemous words negatively affecting retrieval.…”

“…However the use of BERT models directly within the pseudo-relevance feedback mechanism has seen comparatively little use in the literature. The current approaches leveraging the BERT contextualised embeddings for PRF are Neural PRF [18], BERT-QE [37] and CEQE [23].…”

“…Note that, at the time of writing, there is no publicly available implementation of the very recent CEQE [23] approach, and hence we omit it from our experiments. Code to reproduce ColBERT-PRF is available in the PyTerrier_ColBERT repository.…”

“…Indeed, most of the existing embedding-based QE methods [11,17,30,31,36] are based on static embeddings, where a word embedding is always the same within different sentences, and hence they do not address contextualised language models such as BERT. Recently, CEQE [23] was proposed, which makes use of contextualised BERT embeddings for query expansion. The resulting refined query representation is then used for a further round of retrieval using a conventional inverted index.…”

Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls

“…It is clear that integrating PRF signals into deep language models implies a trade-off between effectiveness and efficiency. While current approaches ignored efficiency, the majority still achieved marginal improvements in effective- maintaining efficiency: (i) by concatenating the feedback passages with the original query to form the new queries that contain the relevant signals, (ii) by pre-generating passage collection embeddings and performing PRF in the vector space, because embeddings promise to capture the semantic similarity between terms [10,13,22,36,37,45,56,57],…”