Neural Unsupervised Domain Adaptation in NLP—A Survey

Ramponi, Alan; Plank, Barbara

doi:10.18653/v1/2020.coling-main.603

Cited by 137 publications

(113 citation statements)

References 99 publications

Supporting

Mentioning

108

Contrasting

Order By: Relevance

“…Domain adversarial training is a dominant approach for UDA (Ramponi and Plank, 2020), in-1 https://github.com/ckarouzos/slp_daptmlm spired by the theory for learning from different domains introduced in Ben- David et al (2007David et al ( , 2010. Ganin et al (2016); Ganin and Lempitsky (2015) propose to learn a task while not being able to distinguish if samples come from the source or the target distribution, through use of an adversarial cost.…”

Section: Related Workmentioning

confidence: 99%

UDALM: Unsupervised Domain Adaptation through Language Modeling

Karouzos¹,

Paraskevopoulos²,

Potamianos³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improvement over the state-of-the-art.

show abstract

Section: Related Workmentioning

confidence: 99%

UDALM: Unsupervised Domain Adaptation through Language Modeling

Karouzos¹,

Paraskevopoulos²,

Potamianos³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…It is more practical yet more challenging. For a complete treatment of neural networks and UDA in NLP, refer to (Ramponi and Plank, 2020). Also, we do not treat multilingual work.…”

Section: Applications Of Divergence Measuresmentioning

confidence: 99%

Domain Divergences: A Survey and Empirical Analysis

Kashyap¹,

Hazarika²,

Kan³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Domain divergence plays a significant role in estimating the performance of a model in new domains. While there is a significant literature on divergence measures, researchers find it hard to choose an appropriate divergence for a given NLP application. We address this shortcoming by both surveying the literature and through an empirical study. We develop a taxonomy of divergence measures consisting of three classes -Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. Further, to understand the common use-cases of these measures, we recognise three novel applications -1) Data Selection, 2) Learning Representation, and 3) Decisions in the Wild -and use it to organise our literature. From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and Higher-order measures are more common for 2). To further help researchers choose appropriate measures to predict drop in performance -an important aspect of Decisions in the Wild, we perform correlation analysis spanning 130 domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we consider the current contextual word representations (CWR) and contrast with the older distributed representations. We find that traditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective.

show abstract

“…Unsupervised domain adaptation. In the past few years, there has been considerable interest in unsupervised domain adaptation for cross-domain NLP tasks, including cross-domain sentiment analysis (Ramponi and Plank, 2020). Previous work has focused on minimizing the discrepancy between domains by aligning the output distributions of the source and the target domains.…”

Section: Related Workmentioning

confidence: 99%

“…Maximum Mean Discrepancy (MMD) (Gretton et al, 2012), KL-divergence (Zhuang et al, 2015), Correlation Alignment (CORAL) (Sun and Saenko, 2016), and domain-adversarial learning (Ganin et al, 2016) are among the most widely used methods to learn domain-invariant features. In the same vein, other researchers have adopted self-training approach in order to learn discriminative features of the target domain (Ramponi and Plank, 2020;. The latter approach enables the model to be also trained on some samples of the target domain.…”

Section: Related Workmentioning

confidence: 99%

“…It consists of finetuning the pre-trained language models on large unlabeled corpus from the target domain using the MLM objective. On the other hand, self-training and domain-adversarial learning have been applied successfully to many NLP applications Ramponi and Plank, 2020;Ganin et al, 2016). An effective method that combines domain-adversarial training and self-training is the Adversarial-Learned Loss for Domain Adaptation (ALDA) (Chen et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding

Mekki¹,

Mahdaouy²,

Berrada³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Finetuning deep pre-trained language models has shown state-of-the-art performances on a wide range of Natural Language Processing (NLP) applications. Nevertheless, their generalization performance drops under domain shift. In the case of Arabic language, diglossia makes building and annotating corpora for each dialect and/or domain a more challenging task. Unsupervised Domain Adaptation tackles this issue by transferring the learned knowledge from labeled source domain data to unlabeled target domain data. In this paper, we propose a new unsupervised domain adaptation method for Arabic cross-domain and crossdialect sentiment analysis from Contextualized Word Embedding. Several experiments are performed adopting the coarse-grained and the fine-grained taxonomies of Arabic dialects. The obtained results show that our method yields very promising results and outperforms several domain adaptation methods for most of the evaluated datasets. On average, our method increases the performance by an improvement rate of 20.8% over the zero-shot transfer learning from BERT.

show abstract

Neural Unsupervised Domain Adaptation in NLP—A Survey

Cited by 137 publications

References 99 publications

UDALM: Unsupervised Domain Adaptation through Language Modeling

UDALM: Unsupervised Domain Adaptation through Language Modeling

Domain Divergences: A Survey and Empirical Analysis

Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding

Contact Info

Product

Resources

About