Cross-lingual Transfer of Monolingual Models

Gogoulou, Evangelia; Ekgren, Ariel; Isbister, Tim; Sahlgren, Magnus

doi:10.48550/arxiv.2109.07348

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our experiments build off previous efforts that try to enable crosslingual transfer from pretrained monolingual LLMs to new languages (Artetxe et al, 2018(Artetxe et al, , 2020Tran, 2020;Reimers and Gurevych, 2020;Gogoulou et al, 2021). For example, BERT shows effective but limited transfer when attached to multi-lingual tokenizers (Reimers and Gurevych, 2020).…”

Section: Related Workmentioning

confidence: 90%

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Wu¹,

Papadimitriou²,

Tamkin³

2022

Preprint

View full text Add to dashboard Cite

Little is known about what makes crosslingual transfer hard, since factors like tokenization, morphology, and syntax all change at once between languages. To disentangle the impact of these factors, we propose a set of controlled transfer studies: we systematically transform GLUE tasks to alter different factors one at a time, then measure the resulting drops in a pretrained model's downstream performance. In contrast to prior work suggesting little effect from syntax on knowledge transfer, we find significant impacts from syntactic shifts (3-6% drop), though models quickly adapt with continued pretraining on a small dataset. However, we find that by far the most impactful factor for crosslingual transfer is the challenge of aligning the new embeddings with the existing transformer layers (18% drop), with little additional effect from switching tokenizers (<2% drop) or word morphologies (<2% drop). Moreover, continued pretraining with a small dataset is not very effective at closing this gap-suggesting that new directions are needed for solving this problem.

show abstract

Section: Related Workmentioning

confidence: 90%

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Wu¹,

Papadimitriou²,

Tamkin³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…BERT was trained to model semantics with English sub-word units. As Gogoulou et al [11] show, semantic information can be transferred across language boundaries. FPT based SLT models can leverage this pre-learned semantic information to better model the sign language utterance s.…”

Section: Representation Power Of Neural Sign Language Translation Modelsmentioning

confidence: 99%

“…Artetxe et al [10] show that the attention patterns learned by BERT on one written language transfer to another written language with minimal fine-tuning. Gogoulou et al [11] further illustrate how semantic information is transferred between different languages.…”

Section: Related Workmentioning

confidence: 99%

“…Models such as the well-known "BERT" [8] that are trained in this way can also be adapted as encoders or decoders in an MT model to improve translation performance [9]. Such models can even transfer in few or zero-shot to different languages [10,11], but also across modalities [12], and this gives them great potential in low-resource scenarios. Even if the downstream task is not related to natural language processing, incorporating pretrained language models can provide a powerful initialization for model parameters and improve the performance on the downstream task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation

Coster

Dambre

2022

Information

View full text Add to dashboard Cite

We consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.

show abstract

“…Ogueji et al (2021) and Ogunremi et al (2023), showcase the positive effects of pretraining on closer and related languages to the target language, even if this is less data than larger pretrained models, in part because of the possibility of shared vocabulary (Oladipo et al, 2022). Our experiments build off previous efforts that try to enable crosslingual transfer from pretrained monolingual LLMs to new languages (Artetxe et al, 2018(Artetxe et al, , 2020Tran, 2020;Reimers and Gurevych, 2020;Gogoulou et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies

Wu,

Tamkin,

Papadimitriou

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

When we transfer a pretrained language model to a new language, there are many axes of variation that change at once. To disentangle the impact of different factors like syntactic similarity and vocabulary similarity, we propose a set of controlled transfer studies: we systematically transform the language of the GLUE benchmark, altering one axis of crosslingual variation at a time, and then measure the resulting drops in a pretrained model's downstream performance. We find that models can largely recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens. Moreover, good-quality tokenizers in the transfer language do not make vocabulary alignment easier. Our experiments provide insights into the factors of cross-lingual transfer that researchers should most focus on when designing language transfer scenarios.

show abstract

Cross-lingual Transfer of Monolingual Models

Cited by 3 publications

References 0 publications

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation

Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies

Contact Info

Product

Resources

About