Don’t Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings

Abstract

Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning, where multilingual BERT is fine-tuned on one source language and evaluated on a different target language. However, published results for mBERT zero-shot accuracy vary as much as 17 points on the MLDoc classification task across four papers. We show that the standard practice of using English dev accuracy for model selection in the zero-shot setting makes it difficult to obtain reproducible results on the MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even anti-correlated) with target language accuracy, and zero-shot performance varies greatly at different points in the same fine-tuning run and between different fine-tuning runs. These reproducibility issues are also present for other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We recommend providing oracle scores alongside zero-shot results: still fine-tune using English data, but choose a checkpoint with the target dev set. Reporting this upper bound makes results more consistent by avoiding arbitrarily bad checkpoints.

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…With exactly the same training data, using different random seeds yields a 1-2 accuracy difference of FS-XLT (Figure 1 top). A similar phenomenon has been observed in finetuning monolingual encoders (Dodge et al, 2020) and multilingual encoders with ZS-XLT (Keung et al, 2020a;Wu and Dredze, 2020b;Xia et al, 2020); we show this observation also holds for FS-XLT. The key takeaway is that varying the buckets is a more severe problem.…”

“…For MLDoc, our results are comparable to (Dong and de Melo, 2019;Wu and Dredze, 2019;Eisenschlos et al, 2019). It is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a). For MARC, our zero-shot results are worse than Keung et al (2020b)'s who use the dev set of each target language for model selection while we use EN dev, following the common true ZS-XLT setup.…”

“…A widely explored transfer scenario is zero-shot crosslingual transfer (Pires et al, 2019;Conneau and Lample, 2019;Artetxe and Schwenk, 2019), where a pretrained encoder is finetuned on abundant task data in the source language (e.g., English) and then directly evaluated on target-language test data, achieving surprisingly good performance (Wu and Dredze, 2019;Hu et al, 2020). However, there is evidence that zero-shot performance reported in the literature has large variance and is often not reproducible (Keung et al, 2020a;Rios et al, 2020); the results in languages distant from English fall far short of those similar to English (Hu et al, 2020;Liang et al, 2020). Lauscher et al (2020) stress the importance of few-shot crosslingual transfer instead, where the encoder is first finetuned on a source language and then further finetuned with a small amount (10-100) of examples (few shots) of the target language.…”

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

“…To choose the final model, we use the scores on the English development data. We are aware that this was recently shown to be sub-optimal in some settings (Keung et al, 2020), however there is no clear solution on how to circumvent this in a pure zero-shot cross-lingual setup (i.e. without assuming any target language target task annotation data).…”

AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER

“…Following Keung et al (2020), in all experiments the other hyper-parameters are tuned on each target language dev set. We train all models for 10 epochs and choose the best model checkpoint with the target dev set.…”