Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation

Abstract

The overreliance on large parallel corpora significantly limits the applicability of machine translation systems to the majority of language pairs. Back-translation has been dominantly used in previous approaches for unsupervised neural machine translation, where pseudo sentence pairs are generated to train the models with a reconstruction loss. However, the pseudo sentences are usually of low quality as translation errors accumulate during training. To avoid this fundamental issue, we propose an alternative but more effective approach, extract-edit, to extract and then edit real sentences from the target monolingual corpora. Furthermore, we introduce a comparative translation loss to evaluate the translated target sentences and thus train the unsupervised translation systems. Experiments show that the proposed approach consistently outperforms the previous state-of-the-art unsupervised machine translation systems across two benchmarks (English-French and English-German) and two low-resource language pairs (English-Romanian and English-Russian) by more than 2 (up to 3.63) BLEU points. Recently, neural-based methods (Chu et al., 2016;Grover and Mitra, 2017;Grégoire and Langlais, 2018) aim to select potential parallel sentences

Unsupervised Neural Machine Translation with Future Rewarding

“…Our method is compared with several previous unsupervised NMT systems (Artetxe et al, 2018b;4 https://github.com/artetxem/vecmap Lample et al, 2018a,b;Yang et al, 2018;Wu et al, 2019;Song et al, 2019). Although, Song et al (2019) have achieved comparable results with supervised NMT systems with larger monolingual data (Wikipedia data) and bigger model 5 , we still list the results that obtained with the same data and model as ours for fair comparison.…”

A Retrieve-and-Rewrite Initialization Method for Unsupervised Machine Translation

“…But they use much more monolingual data from Wikipedia than previous work and this paper. We must also mention the work of Wu et al (2019). They similarly use retrieval and rewriting framework for unsupervised MT.…”

Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring

“…Artetxe et al (2018a) combined back-translation and MERT (Och, 2003) to iteratively refine the SMT model. Wu et al (2019) proposed to discard back-translation. Instead, they extracted and edited the nearest sentences in the target language to construct pseudo-parallel data, which was used as a supervision signal.…”

“…Building an effective model on low resource data or even in an unsupervised way is always an interesting and challenging research topic (Gu et al, 2018;Radford et al, 2016;Lee et al, 2019). Recently, unsupervised MT (Artetxe et al, 2018b,a;Conneau et al, 2018;Lample et al, 2018b;Wu et al, 2019), which can immensely reduce the reliance on parallel corpora, has been gaining more and more interest.…”