GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Wang, Kexin; Nandan, Thakur,; Reimers, Nils; Gurevych, Iryna

doi:10.18653/v1/2022.naacl-main.168

Cited by 39 publications

(33 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In NLP, Wang et al [171] used Generative Pseudo Labeling (GPL) for query-passage extraction purposes: where they retrieved positive passages from labeled data and applied that model for retrieving negative passages in target data. Thereafter, they used Margin-MSE loss which helped the cross-encoder to soft-label query-passage pairs effectively.…”

Section: ) Pseudo-semi-supervised Domain Adaptationmentioning

confidence: 99%

Domain Adaptation: Challenges, Methods, Datasets, and Applications

et al. 2023

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) trained on one dataset (source domain) do not perform well on another set of data (target domain), which is different but has similar properties as the source domain. Domain Adaptation (DA) strives to alleviate this problem and has great potential in its application in practical settings, real-world scenarios, industrial applications and many data domains. Various DA methods aimed at individual data domains have been reported in the last few years; however, there is no comprehensive survey that encompasses all these data domains, focuses on the datasets available, the methods relevant to each domain, and importantly the applications and challenges. To that end, this survey paper discusses how DA can help DNNs work efficiently in these settings by reviewing DA methods and techniques. We have considered five data domains: computer vision, natural language processing, speech, time-series, and multi-modal data. We present a comprehensive taxonomy, including the methods, datasets, challenges, and applications corresponding to each domain. Our goal is to discuss industrial use cases and DA implementation for those. Our final aim is to provide future research directions based on evolving methods and results, the datasets used, and industrial applications.

show abstract

Section: ) Pseudo-semi-supervised Domain Adaptationmentioning

confidence: 99%

Domain Adaptation: Challenges, Methods, Datasets, and Applications

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Creating datasets is very expensive but often necessary for domain adaptation. A growing trend is the generation of synthetic QA datasets from models [137] or unstructured text using different techniques such as ICT [138], GPL [139], GenQ [140], Promptagator [141], COCO-DR [142]. Some other technics like natural language augmentation [143] aims at enriching existing datasets for a more robust training through transformation and data filtering.…”

Section: Big Bench Datasets Formentioning

confidence: 99%

Complex QA and language models hybrid architectures, Survey

Daull¹,

Bellot²,

Bruno³

et al. 2023

Preprint

View full text Add to dashboard Cite

This paper provides a survey of the state of the art of hybrid language models architectures and strategies for "complex" question-answering (QA, CQA, CPS). Very large language models are good at leveraging public data on standard problems but once you want to tackle more specific complex questions or problems you may need specific architecture, knowledge, skills, tasks, methods, sensitive data, performance, human approval and versatile feedback... This survey extends findings from the robust community edited research papers BIG, BLOOM and HELM which open source, benchmark and analyze limits and challenges of large language models in terms of tasks complexity and strict evaluation on accuracy (e.g. fairness, robustness, toxicity, ...). It identifies the key elements used with Large Language Models (LLM) to solve complex questions or problems. Recent projects like ChatGPT and GALACTICA have allowed non-specialists to grasp the great potential as well as the equally strong limitations of language models in complex QA. Hybridizing these models with different components could allow to overcome these different limits and go much further. We discuss some challenges associated with complex QA, including domain adaptation, decomposition and efficient multi-step QA, long form QA, non-factoid QA, safety and multi-sensitivity data protection, multimodal search, hallucinations, QA explainability and truthfulness, time dimension. Therefore we review current solutions and promising strategies, using elements such as hybrid LLM architectures, human-in-the-loop reinforcement learning, prompting adaptation, neuro-symbolic and structured knowledge grounding, program synthesis, and others. We analyze existing solutions and provide an overview of the current research and trends in the area of complex QA.

show abstract

“…In addition, to prompt-based generation of training data, there are multiple proposals for self-supervised adaptation of out-of-domain models using generative pseudo-labeling [22,38,51]. To this end, questions or queries are generated using a pretrained seq2seq model (though an LLMs can be used as well) and negative examples are mined using either BM25 or an out-of-domain retriever or ranker.…”

Section: Related Workmentioning

confidence: 99%

InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers

Boytsov¹,

Patel²,

Sourabh³

et al. 2023

Preprint

View full text Add to dashboard Cite

We carried out a reproducibility study of InPars recipe for unsupervised training of neural rankers [4]. As a by-product of this study, we developed a simple-yet-effective modification of InPars, which we called InPars-light. Unlike InPars, InPars-light uses only a freely available language model BLOOM and 7x-100x smaller ranking models. On all five English retrieval collections (used in the original InPars study) we obtained substantial (7-30%) and statistically significant improvements over BM25 in nDCG or MRR using only a 30M parameter six-layer MiniLM ranker. In contrast, in the InPars study only a 100x larger MonoT5-3B model consistently outperformed BM25, whereas their smaller MonoT5-220M model (which is still 7x larger than our MiniLM ranker), outperformed BM25 only on MS MARCO and TREC DL 2020. In a purely unsupervised setting, our 435M parameter DeBERTA v3 ranker was roughly at par with the 7x larger MonoT5-3B: In fact, on three out of five datasets, it slightly outperformed MonoT5-3B. Finally, these good results were achieved by re-ranking only 100 candidate documents compared to 1000 used in InPars. We believe that InPars-light is the first truly cost-effective prompt-based unsupervised recipe to train and deploy neural ranking models that outperform BM25. CCS CONCEPTS• Information systems → Retrieval models and ranking.

show abstract

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Cited by 39 publications

References 11 publications

Domain Adaptation: Challenges, Methods, Datasets, and Applications

Domain Adaptation: Challenges, Methods, Datasets, and Applications

Complex QA and language models hybrid architectures, Survey

InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers

Contact Info

Product

Resources

About