PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models

In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improvement over the state-of-the-art.

show abstract

“…This source-only model even surpasses state-of-the-art methods developed for UDA, e.g. R-PERL (Ben-David et al, 2020).…”

Section: Comparison To State-of-the-artmentioning

confidence: 89%

UDALM: Unsupervised Domain Adaptation through Language Modeling

Karouzos¹,

Paraskevopoulos²,

Potamianos³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…• Two-stage fine-tuning introduces an intermediate supervised training stage between pre-training and fine-tuning Arase and Tsujii, 2019;Pruksachatkun et al, 2020;Glavaš and Vulić, 2020). Ben-David et al (2020) propose a pivot-based variant of MLM to fine-tune BERT for domain adaptation.…”

Section: Fine-tuning Bertmentioning

confidence: 99%

A Primer in BERTology: What We Know About How BERT Works

Rogers

Kovaleva

Rumshisky

2020

Transactions of the Association for Computational Linguistics

935

590

View full text Add to dashboard Cite

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.

show abstract

“…Work on the intersection of data-centric and model-centric methods can be plentiful. It currently includes combining semi-supervised objectives with an adversarial loss (Lim et al, 2020;Alam et al, 2018b), combining pivot-based approaches with pseudo-labeling (Cui and Bollegala, 2019) and very recently with contextualized word embeddings (Ben-David et al, 2020), and combining multi-task approaches with domain shift (Jia et al, 2019), multi-task learning with pseudo-labeling (multi-task tritraining) (Ruder and Plank, 2018), and adaptive ensembling (Desai et al, 2019), which uses a studentteacher network with a consistency-based self-ensembling loss and a temporal curriculum. They apply adaptive ensembling to study temporal and topic drift in political data classification (Desai et al, 2019).…”

Section: Hybrid Approachesmentioning

confidence: 99%

Neural Unsupervised Domain Adaptation in NLP—A Survey

Ramponi¹,

Plank²

2020

Proceedings of the 28th International Conference on Computational Linguistics

148

108

View full text Add to dashboard Cite

Deep neural networks excel at learning from labeled data and achieve state-of-the-art results on a wide array of Natural Language Processing tasks. In contrast, learning from unlabeled data, especially under domain shift, remains a challenge. Motivated by the latest advances, in this survey we review neural unsupervised domain adaptation techniques which do not require labeled target domain data. This is a more challenging yet a more widely applicable setup. We outline methods, from early traditional non-neural methods to pre-trained model transfer. We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention. Lastly, we outline future directions, particularly the broader need for out-of-distribution generalization of future NLP. 1

show abstract

PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models

Cited by 40 publications

References 28 publications

UDALM: Unsupervised Domain Adaptation through Language Modeling

UDALM: Unsupervised Domain Adaptation through Language Modeling

A Primer in BERTology: What We Know About How BERT Works

Neural Unsupervised Domain Adaptation in NLP—A Survey

Contact Info

Product

Resources

About