Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information

Abstract

We investigate the following question for machine translation (MT): can we develop a single universal MT model to serve as the common seed and obtain derivative and improved models on arbitrary language pairs? We propose mRASP, an approach to pre-train a universal multilingual neural machine translation model. Our key idea in mRASP is its novel technique of random aligned substitution, which brings words and phrases with similar meanings across multiple languages closer in the representation space. We pre-train a mRASP model on 32 language pairs jointly with only public datasets. The model is then fine-tuned on downstream language pairs to obtain specialized MT models. We carry out extensive experiments on 42 translation directions across a diverse settings, including low, medium, rich resource, and as well as transferring to exotic language pairs. Experimental results demonstrate that mRASP achieves significant performance improvement compared to directly training on those target pairs. It is the first time to verify that multiple lowresource language pairs can be utilized to improve rich resource MT. Surprisingly, mRASP is even able to improve the translation quality on exotic languages that never occur in the pretraining corpus. Code, data, and pre-trained models are available at https://github. com/linzehui/mRASP. * Equal contribution. The work was done when the first author was an intern at ByteDance.

Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation

“…Some explores data symmetry (Freitag and Firat, 2020;Birch et al, 2008;Lin et al, 2019). Zero-shot translation in severely low resource settings exploits the massive multilinguality, cross-lingual transfer, pretraining, iterative backtranslation and freezing subnetworks (Lauscher et al, 2020;Nooralahzadeh et al, 2020;Pfeiffer et al, 2020;Baziotis et al, 2020;Chronopoulou et al, 2020;Lin et al, 2020;Thompson et al, 2018;Luong et al, 2014;Dou et al, 2020).…”

“…We find that using many languages that are distant to the target low resource language may produce marginal improvements, if not negative impact. Indeed, existing literature on zero-shot translation also suffers from the limitation of linguistic distance between the source languages and the target language (Lauscher et al, 2020;Lin et al, 2020;Pfeiffer et al, 2020). We therefore rank and select the top few source languages that are closer to the target low resource language using the two metrics below.…”

“…Instead, we examine ways to pick useful source languages from 124 source languages in a principled fashion. Secondly, most works require at least 4,000 lines of low resource data (Lin et al, 2020;Qi et al, 2018;Zhou et al, 2018a); we use only ∼1,000 lines of low resource data to simulate real-life situation of having ex-tremely small seed target translation. Thirdly, many works use rich resource languages as hypothetical low resource languages.…”

Active Learning for Massively Parallel Translation of Constrained Text into Low Resource Languages

“…Some use data selection for active learning (Eck et al, 2005). Some use as few as ∼4,000 lines (Lin et al, 2020;Qi et al, 2018) and ∼1,000 lines (Zhou and Waibel, 2021) of data. Some do not use low resource data (Neubig and Hu, 2018;Karakanta et al, 2018).…”

“…Given a closed text that has many existing translations in different languages, we are interested in translating it into a severely low resource language well. Researchers recently have shown achievements in translation using very small seed parallel corpora in low resource languages (Lin et al, 2020;Qi et al, 2018;Zhou et al, 2018a). Construction methods of such seed corpora are therefore pivotal in translation performance.…”