How to Adapt Your Pretrained Multilingual Model to 1600 Languages

Abstract

Pretrained multilingual models (PMMs) enable zero-shot learning via cross-lingual transfer, performing best for languages seen during pretraining. While methods exist to improve performance for unseen languages, they have almost exclusively been evaluated using amounts of raw text only available for a small fraction of the world's languages. In this paper, we evaluate the performance of existing methods to adapt PMMs to new languages using a resource available for over 1600 languages: the New Testament. This is challenging for two reasons: (1) the small corpus size, and (2) the narrow domain. While performance drops for all approaches, we surprisingly still see gains of up to 17.69% accuracy for part-ofspeech tagging and 6.29 F1 for NER on average over all languages as compared to XLM-R. Another unexpected finding is that continued pretraining, the simplest approach, performs best. Finally, we perform a case study to disentangle the effects of domain and size and to shed light on the influence of the finetuning source language.

Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

“…2 Among the 7,000 languages in the world, mBERT only covers about 1% of the languages while Wikipedia and Com-monCrawl, the two most common resources used for pretraining and adaptation, only contain textual data from 4% of the languages (often in quite small quantities, partially because language IDs are difficult to obtain for low-resource languages (Caswell et al, 2020)). Ebrahimi and Kann (2021) show that continued pretraining of multilingual models on a small amount of Bible data can significantly improve the performance of uncovered languages. Although the Bible has much better language coverage of 23%, its relatively small data size and constrained domain limits its utility (see § 6)-and 70% of the world's languages do not even have this resource.…”

“…MLM Continued pretraining on monolingual text D T mono = {x T i } i in the target language (Howard and Ruder, 2018;Gururangan et al, 2020) using a masked language model (MLM) objective has proven effective for adapting models to the target language (Pfeiffer et al, 2020). Notably, Ebrahimi and Kann (2021) show that using as little as several thousand sentences can significantly improve the model's performance on target languages not covered during pretraining.…”

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

“…If on the other hand retraining from scratch is not feasible, one option is to add new subword units for the underresourced/oversegmented languages. Wang et al (2020b) and Chau et al (2020) both propose such additions with randomly initialized embeddings, but these approaches did not perform well when studied by Ebrahimi and Kann (2021); extending the idea, propose to use information about existing subword units to estimate embeddings instead of initializing newly added units randomly (similar to Salesky et al (2018)). A different option is proposed by Wang et al (2021b), who instead force the model to use (already existing) smaller subword units in highresource languages like English to make the segmentations across languages more similar and thus aid transfer-thus avoiding the complete retraining that comes with changing the segmentation method or allocation.…”

Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

“…Furthermore, due to data sparsity, monolingual pretrained models are not likely to obtain good results for many lowresource languages. In those cases, multilingual models can zero-shot learn for unseen languages with an above-chance performance, which can be further improved via model adaptation with targetlanguage text (Wang et al, 2020a), even for limited amounts (Ebrahimi and Kann, 2021). However, it is poorly understood how the number of pretraining languages influences performance in those cases.…”