Don’t Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings

Abstract

Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning, where multilingual BERT is fine-tuned on one source language and evaluated on a different target language. However, published results for mBERT zero-shot accuracy vary as much as 17 points on the MLDoc classification task across four papers. We show that the standard practice of using English dev accuracy for model selection in the zero-shot setting makes it difficult to obtain reproducible results on the MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even anti-correlated) with target language accuracy, and zero-shot performance varies greatly at different points in the same fine-tuning run and between different fine-tuning runs. These reproducibility issues are also present for other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We recommend providing oracle scores alongside zero-shot results: still fine-tune using English data, but choose a checkpoint with the target dev set. Reporting this upper bound makes results more consistent by avoiding arbitrarily bad checkpoints.

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…With exactly the same training data, using different random seeds yields a 1-2 accuracy difference of FS-XLT (Figure 1 top). A similar phenomenon has been observed in finetuning monolingual encoders (Dodge et al, 2020) and multilingual encoders with ZS-XLT (Keung et al, 2020a;Wu and Dredze, 2020b;Xia et al, 2020); we show this observation also holds for FS-XLT. The key takeaway is that varying the buckets is a more severe problem.…”

“…For MLDoc, our results are comparable to (Dong and de Melo, 2019;Wu and Dredze, 2019;Eisenschlos et al, 2019). It is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a). For MARC, our zero-shot results are worse than Keung et al (2020b)'s who use the dev set of each target language for model selection while we use EN dev, following the common true ZS-XLT setup.…”

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…With exactly the same training data, using different random seeds yields a 1-2 accuracy difference of FS-XLT (Figure 1 top). A similar phenomenon has been observed in finetuning monolingual encoders (Dodge et al, 2020) and multilingual encoders with ZS-XLT (Keung et al, 2020a;Wu and Dredze, 2020b;Xia et al, 2020); we show this observation also holds for FS-XLT. The key takeaway is that varying the buckets is a more severe problem.…”

“…For MLDoc, our results are comparable to (Dong and de Melo, 2019;Wu and Dredze, 2019;Eisenschlos et al, 2019). It is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a). For MARC, our zero-shot results are worse than Keung et al (2020b)'s who use the dev set of each target language for model selection while we use EN dev, following the common true ZS-XLT setup.…”

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…For MLDoc, our results are comparable to Dong and de Melo (2019); Wu and Dredze (2019); Eisenschlos et al (2019). However, it is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a).…”

Predicate-Argument Structure Divergences in Chinese and English Parallel Sentences and their Impact on Language Transfer

“…While AMR structures aligned well in many cases, non-local structural divergences were observed, particularly between English and Czech, where 53In another study, [Sulem et al, 2015] compared English and French translations using Universal Conceptual Cognitive Annotation [Abend and Rappoport, 2013, UCCA], another semantic representation formalism. UCCA graphs, which represent meaning as directed acyclic graphs (DAGs), demonstrated greater stability across English-French translations than AMR graphs, corroborating other findings on compatibility between these typologically similar languages [Pires et al, 2019, Keung et al, 2020, Blloshmi et al, 2020]. However, a systematic explanation for this phenomenon is still lacking.…”

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…With exactly the same training data, using different random seeds yields a 1-2 accuracy difference of FS-XLT (Figure 1 top). A similar phenomenon has been observed in finetuning monolingual encoders (Dodge et al, 2020) and multilingual encoders with ZS-XLT (Keung et al, 2020a;Wu and Dredze, 2020b;Xia et al, 2020); we show this observation also holds for FS-XLT. The key takeaway is that varying the buckets is a more severe problem.…”

“…For MLDoc, our results are comparable to (Dong and de Melo, 2019;Wu and Dredze, 2019;Eisenschlos et al, 2019). It is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a). For MARC, our zero-shot results are worse than Keung et al (2020b)'s who use the dev set of each target language for model selection while we use EN dev, following the common true ZS-XLT setup.…”

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…For MLDoc, our results are comparable to Dong and de Melo (2019); Wu and Dredze (2019); Eisenschlos et al (2019). However, it is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a).…”

Predicate-Argument Structure Divergences in Chinese and English Parallel Sentences and their Impact on Language Transfer

“…While AMR structures aligned well in many cases, non-local structural divergences were observed, particularly between English and Czech, where 53In another study, [Sulem et al, 2015] compared English and French translations using Universal Conceptual Cognitive Annotation [Abend and Rappoport, 2013, UCCA], another semantic representation formalism. UCCA graphs, which represent meaning as directed acyclic graphs (DAGs), demonstrated greater stability across English-French translations than AMR graphs, corroborating other findings on compatibility between these typologically similar languages [Pires et al, 2019, Keung et al, 2020, Blloshmi et al, 2020]. However, a systematic explanation for this phenomenon is still lacking.…”

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…With exactly the same training data, using different random seeds yields a 1-2 accuracy difference of FS-XLT (Figure 1 top). A similar phenomenon has been observed in finetuning monolingual encoders (Dodge et al, 2020) and multilingual encoders with ZS-XLT (Keung et al, 2020a;Wu and Dredze, 2020b;Xia et al, 2020); we show this observation also holds for FS-XLT. The key takeaway is that varying the buckets is a more severe problem.…”

“…For MLDoc, our results are comparable to (Dong and de Melo, 2019;Wu and Dredze, 2019;Eisenschlos et al, 2019). It is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a). For MARC, our zero-shot results are worse than Keung et al (2020b)'s who use the dev set of each target language for model selection while we use EN dev, following the common true ZS-XLT setup.…”

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

“…For MLDoc, our results are comparable to Dong and de Melo (2019); Wu and Dredze (2019); Eisenschlos et al (2019). However, it is worth noting that reproducing the exact results is challenging, as suggested by Keung et al (2020a).…”

Predicate-Argument Structure Divergences in Chinese and English Parallel Sentences and their Impact on Language Transfer

“…While AMR structures aligned well in many cases, non-local structural divergences were observed, particularly between English and Czech, where 53In another study, [Sulem et al, 2015] compared English and French translations using Universal Conceptual Cognitive Annotation [Abend and Rappoport, 2013, UCCA], another semantic representation formalism. UCCA graphs, which represent meaning as directed acyclic graphs (DAGs), demonstrated greater stability across English-French translations than AMR graphs, corroborating other findings on compatibility between these typologically similar languages [Pires et al, 2019, Keung et al, 2020, Blloshmi et al, 2020]. However, a systematic explanation for this phenomenon is still lacking.…”