Adversarial Retriever-Ranker for dense text retrieval

Abstract

Current dense text retrieval models face two typical challenges. First, it adopts a siamese dual-encoder architecture to encode query and document independently for fast indexing and searching, whereas neglecting the finer-grained term-wise interactions. This results in a sub-optimal recall performance. Second, it highly relies on a negative sampling technique to build up the negative documents in its contrastive loss. To address these challenges, we present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker. The two models are jointly optimized according to a minimax adversarial objective: the retriever learns to retrieve negative documents to cheat the ranker, while the ranker learns to rank a collection of candidates including both the ground-truth and the retrieved ones, as well as providing progressive direct feedback to the dual-encoder retriever. Through this adversarial game, the retriever gradually produces harder negative documents to train a better ranker, whereas the cross-encoder ranker provides progressive feedback to improve retriever. We evaluate AR2 on three benchmarks. Experimental results show that AR2 consistently and significantly outperforms existing dense retriever methods and achieves new state-of-the-art results on all of them. This includes the improvements on Natural Questions R@5 to 77.9% (+2.1%), TriviaQA R@5 to 78.2% (+1.4%), and MS-MARCO MRR@10 to 39.5% (+1.3%). We will make our code, models, and data publicly available.

Time and frequency dynamic connectedness between cryptocurrencies and financial assets in China

“…Therefore, using the knowledge learned by the ranker to guide the retriever is an effective approach. The Adversarial Retriever-Ranker(AR2) [34] model introduces the generative adversarial network into information retrieval, drawing inspiration from the IRGAN framework [31]. The dual-encoder retriever serves as the generator, retrieving hard negative samples to confuse the ranker.…”

Boosted Dense Retriever

“…It also strongly outperforms a dimensionally-matched DPR, by 3% on MS-MARCO, and 1% on NQ in R@100, demonstrating DrBoost's ability to learn high-quality, compact embeddings. We also quote recent state-of-the-art results, which generally achieve stronger exact search results (AR2, Zhang et al, 2021). Our emphasis, however, is on comparing iteratively-sampled negatives to boosting, and we note that state-ofthe-art approaches generally use larger models and more complex training strategies than the "inner loop" BERT-base DPR we report here.…”

ConTextual Masked Auto-Encoder for Dense Passage Retrieval

“…Fine-tuning Many attempts have explored to improve fine-tuning peformance, such as mining hard negatives (Xiong et al 2020;Zhan et al 2021), late interaction (Khattab and Zaharia 2020), distill knowledge from a strong teacher (Lin, Yang, and Lin 2021;Santhanam et al 2021), query clustering (Hofstätter et al 2021), data augmentation (Qu et al 2020) and jointly optimize retriever and re-ranker (Ren et al 2021b;Zhang et al 2022Zhang et al , 2021 .…”