Improving Neural Machine Translation Models with Monolingual Data

Abstract

Neural Machine Translation (NMT) has obtained state-of-the art performance for several language pairs, while only using parallel data for training. Targetside monolingual data plays an important role in boosting fluency for phrasebased statistical machine translation, and we investigate the use of monolingual data for NMT. In contrast to previous work, which combines NMT models with separately trained language models, we note that encoder-decoder NMT architectures already have the capacity to learn the same information as a language model, and we explore strategies to train with monolingual data without changing the neural network architecture. By pairing monolingual training data with an automatic backtranslation, we can treat it as additional parallel training data, and we obtain substantial improvements on the WMT 15 task English↔German (+2.8-3.7 BLEU), and for the low-resourced IWSLT 14 task Turkish→English (+2.1-3.4 BLEU), obtaining new state-of-the-art results. We also show that fine-tuning on in-domain monolingual and parallel data gives substantial improvements for the IWSLT 15 task English→German.

Neural machine translation for low-resource languages without parallel corpora

“…Previous work combines NMT models with separately trained language models (Gülçehre et al 2015). Sennrich et al (2015) show that target-side monolingual data can greatly enhance the decoder model. They do not propose any changes in the network architecture, but rather pair monolingual data with automatic back-translations and treat it as additional training data.…”

The JHU Machine Translation Systems for WMT 2017

“…To investigate the effectiveness of incorporating monolingual information with back-translation (Sennrich et al, 2016b), we continued training on top of the base system to build another system (labeled back-trans below) that has some exposure to the monolingual data. Due to the time and hardware constraints, we only took a random sample of 2 million sentences from news crawl 2016 monolingual corpus and 1.5 million sentences from preprocessed CWMT Chinese monolingual corpus from our syntax-based system run and backtranslated them with our trained base system.…”

Employing Pivot Language Technique Through Statistical and Neural Machine Translation Frameworks : The Case of Under-Resourced Persian-Spanish Language Pair

“…Researchers such as Gulccehre et al (2015) proposed to incorporate the target side monolingual corpora as the language model for NMT [16]. As Sennrich et al (2016) pairs the target monolingual corpora with its corresponding translations then merges them with parallel data for retraining the source-target model [25].…”