UDALM: Unsupervised Domain Adaptation through Language Modeling

Abstract

In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improvement over the state-of-the-art.

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

“…Zero-Shot Models: We apply supervised training on MS MARCO or PAQ and evaluate the trained retrievers on the target datasets. Previous Domain Adaptation Methods: We include two previous unsupervised domain adaptation methods, UDALM (Karouzos et al, 2021) and MoDIR Thakur et al (2021b) to train QGen models with the default setting. Co-12 https://github.com/UKPLab/beir sine similarity is used and the models are fine-tuned for 1 epoch with MNRL.…”

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

“…The evaluation results on the original BioASQ and TREC-COVID are available at Appendix C. Evaluation is done using nDCG@10. Previous Domain Adaptation Methods: We include two previous unsupervised domain adaptation methods, UDALM (Karouzos et al, 2021) and MoDIR . uses the default setting in original paper, where 15% tokens in a text are sampled to be masked and are needed to be predicted.…”

“…MoDIR trains models by generating domain invariant representations to attack a domain classifier. However, as argued in Karouzos et al (2021), DAT trains models by minimizing the distance between representations from different domains and such learning objective can result in bad embedding space and unstable performance. For sentiment classification, Karouzos et al (2021) proposes UDALM based on multiple stages of training.…”

“…However, as argued in Karouzos et al (2021), DAT trains models by minimizing the distance between representations from different domains and such learning objective can result in bad embedding space and unstable performance. For sentiment classification, Karouzos et al (2021) proposes UDALM based on multiple stages of training. UDALM first applies MLM training on the target domain; and it then applies multi-task learning on the target domain with MLM and on the source domain with a supervised objective.…”

Few-shot Mining of Naturally Occurring Inputs and Outputs

“…While we use lessons about careful design choices, our basic models use standard fine-tuning for simplicity. Efforts on the data side have focused on intermediate fine-tuning either by using unlabeled target domain data (Karouzos et al, 2021;Gururangan et al, 2020) or via labeled data from other tasks (Phang et al, 2018;Aghajanyan et al, 2021;Vu et al, 2020).…”