Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER

Abstract

Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated stateof-the-art performance on various NLP tasks. Recent work with the multilingual version of BERT has shown that the model performs very well in cross-lingual settings, even when only labeled English data is used to finetune the model. We improve upon multilingual BERT's zero-resource cross-lingual performance via adversarial learning. We report the magnitude of the improvement on the multilingual ML-Doc text classification and CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that language-adversarial training encourages BERT to align the embeddings of English documents and their translations, which may be the cause of the observed performance gains.

Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification

“…Artetxe et al [1] pretrain a massively multilingual sequence-to-sequence neural MT model, invoking its encoder as a multilingual text representation used for fine-tuning on downstream tasks. Keung et al [10] apply languageadversarial learning into Multilingual BERT during fine-tuning with unlabeled data. We also considered Multilingual BERT and itself with self-learning and adversarial training respectively as our baselines.…”

Margin-aware Unsupervised Domain Adaptation for Cross-lingual Text Labeling

“…They leverage the benefit of contextualized word embeddings by using multilingual BERT (Devlin et al, 2019) as the feature generator, and adopt the GAN framework (Goodfellow et al, 2014) to align the features from the two domains. Keung et al (2019) show significant improvement over the baseline where the pretrained multilingual BERT is finetuned on the English data alone and testing on the same tasks in other languages. However, Keung et al (2019), as well as the works mentioned above, are inspired by the pioneering work of Ben-David et al (2010), which only rigorously studies domain adaptation in the setting of binary classification; there is a lack of theoretical guarantees when it comes to multiclass classification.…”

“…Unsupervised domain adaptation provides an appealing solution to many applications where direct access to a massive amount of labeled data is prohibitive or very costly (Sun and Saenko, 2014;Vazquez et al, 2013;Stark et al, 2010;Keung et al, 2019). For example, we often have sufficient labeled data for English, while very limited or even no labeled data are available for many other languages.…”

“…Many recent successes in unsupervised domain adaptation have been achieved by learning domain invariant features that are simultaneously being discriminative to the task in the source domain (Chen et al, 2018;Ganin and Lempitsky, 2014;Ganin et al, 2016;Tzeng et al, 2017). Following this line, Keung et al (2019) propose a language-adversarial training approach for cross-lingual document classification and NER. They leverage the benefit of contextualized word embeddings by using multilingual BERT (Devlin et al, 2019) as the feature generator, and adopt the GAN framework (Goodfellow et al, 2014) to align the features from the two domains.…”

“…In this work, we are instead motivated by a recent work (Zhang et al, 2019b) that focuses on the theoretical analysis of unsupervised domain adaption for multiclass classification and provides explicit guidance for algorithm design. Instead of training a discriminator that predicts if the representations are from the source domain or the target domain (Keung et al, 2019;Ganin and Lempitsky, 2014;Ganin et al, 2016), Zhang et al (2019b) proposes to optimize an auxiliary classifier which, together with the classifier, minimizes the discrepancy between the two domains via adversarial training. We apply this approach to cross-lingual text labeling tasks, which, as demonstrated in Section 4, outperforms Keung et al (2019) by a large margin.…”