Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback

Abstract

Dense retrieval systems conduct first-stage retrieval using embedded representations and simple similarity metrics to match a query to documents. Its effectiveness depends on encoded embeddings to capture the semantics of queries and documents, a challenging task due to the shortness and ambiguity of search queries. This paper proposes ANCE-PRF, a new query encoder that uses pseudo relevance feedback (PRF) to improve query representations for dense retrieval. ANCE-PRF uses a BERT encoder that consumes the query and the top retrieved documents from a dense retrieval model, ANCE, and it learns to produce better query embeddings directly from relevance labels. It also keeps the document index unchanged to reduce overhead. ANCE-PRF significantly outperforms ANCE and other recent dense retrieval systems on several datasets. Analysis shows that the PRF encoder effectively captures the relevant and complementary information from PRF documents, while ignoring the noise with its learned attention mechanism. CCS CONCEPTS• Information systems → Retrieval models and ranking.

LoL: A Comparative Regularization Loss over Query Reformulation Losses for Pseudo-Relevance Feedback

“…Dense retrieval has made great progress in recent years. Since dense retrievers [23,24,56] use embedding vectors to represent queries and documents, a few methods [28,29,54,59] have been studied to integrate pseudo-relevance information into reformulated query vectors. ColBERT-PRF [54] first verified the effectiveness of PRF in multi-representation dense retrieval [24].…”

“…[28] investigated two simple methods, Average and Rocchio [50], to utilize feedback documents in single-representation dense retrievers (e.g., ANCE [56]) without introducing new neural models or further training. Instead of refining the query vector heuristically, ANCE-PRF [59] uses RoBERTa [32] to consume the original query and the topretrieved documents from ANCE [56]. Keeping the document index unchanged, ANCE-PRF is trained end-to-end with relevance labels and learns to optimize the query vector by exploiting the relevant information in the feedback documents.…”

“…Early PRF was widely studied for sparse retrieval like vector space models [50], probabilistic models [49], and language modeling methods [21,25,26,35,53,61]. Recently, some work has shifted to apply PRF in dense retrieval of single-representation [28,29,59] and multi-representation [54].…”

“…Although PRF methods are generally accepted to improve average retrieval effectiveness [5,6,34,59], their performance is sometimes inferior to the original query [40,53,64]. One of the causes for this robustness problem is query drift: the topic of the revision drifts away from the original intent [40].…”

“…Previous studies cope with query drift mainly by adding preprocessing or post-processing [2,7,33,40] or leveraging state-ofthe-art pre-trained language models [54,59]. However, additional processing brings more computational cost, and pre-trained language models may not necessarily learn to suppress irrelevant information for retrieval without particular supervision [36].…”