Multilingual Denoising Pre-training for Neural Machine Translation

Abstract

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART—a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective (Lewis et al., 2019 ). mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, whereas previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine-tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task- specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show that it enables transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training. 1

Multi-task Learning for Multilingual Neural Machine Translation

“…Moreover, it is less applicable to lowresource language pairs without adequate bitext data. Self-supervised pre-training approaches (Radford et al, 2018;Devlin et al, 2019;Conneau and Lample, 2019;Lewis et al, 2019;Liu et al, 2020), which train the model with denoising learning objectives on the large-scale monolingual data, have achieved remarkable performances in many NLP applications. However, catastrophic forgetting effect (Thompson et al, 2019), where finetuning on a task leads to degradation on the main task, limits the success of continuing training NMT on models pre-trained with monolingual data.…”

“…Self-supervised Learning This work is motivated by the recent success of self-supervised learning for NLP applications (Radford et al, 2018;Devlin et al, 2019;Lample et al, 2018a,b;Conneau and Lample, 2019;Lewis et al, 2019;Liu et al, 2020). Different denoising objectives have been designed to train the neural networks on large-scale unlabeled text.…”

Collection of Meta Information with User-Generated Question Answer Pairs and its Reflection for Improving Expressibility in Response Generation

“…For future work, we want to improve the quality of our generation models since there seems to be much room for improvement when compared to human performance. It may also be interesting to apply other pre-training methods (Yang et al 2019;Liu et al 2020) as well as to incorporate knowledge of the characters in question (Ghazvininejad et al 2018) in order to enhance the character-ness of the generated utterances. We also want to examine the relationship between the naturalness of a generated response and the degree to which the meta information can be reflected.…”

“…, pre-trained language models are showing promising results in a wide variety of natural language processing tasks(Devlin et al 2019;Radford et al 2019;Yang et al 2019;Liu et al 2020). Such models can more accurately capture the meaning of words depending on the context with a massive amount of training data, enabling them to be applied to finetuning for particular downstream tasks.…”

Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages

“…Another recent approach, mBART (Liu et al, 2020), leverages both monolingual and parallel data and also yields improvements in translation quality for lower-resource languages such as Nepali, Sinhala and Gujarati. 3 While this provides a solution for small quantities of training data or monolingual resources, the extent to which standard BLEU evaluations reflect translation quality is not clear yet, since human evaluation studies are missing.…”