The RWTH Aachen University Supervised Machine Translation Systems for WMT 2018

Abstract

This paper describes the statistical machine translation systems developed at RWTH Aachen University for the German→English, English→Turkish and Chinese→English translation tasks of the EMNLP 2018 Third Conference on Machine Translation (WMT 2018). We use ensembles of neural machine translation systems based on the Transformer architecture. Our main focus is on the German→English task where we scored first with respect to all automatic metrics provided by the organizers. We identify data selection, fine-tuning, batch size and model dimension as important hyperparameters. In total we improve by 6.8% BLEU over our last year's submission and by 4.8% BLEU over the winning system of the 2017 German→English task. In English→Turkish task, we show 3.6% BLEU improvement over the last year's winning system. We further report results on the Chinese→English task where we improve 2.2% BLEU on average over our baseline systems but stay behind the 2018 winning systems.

Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies

“…We replaced non-overlapping tokens in the German source side with the closest Basque token in the cross-lingual word embedding space. The result is, however, worse than not replacing them; we noticed that this subword-by-subword translation produces many Basque phrases with wrong BPE merges (Kim et al, 2018).…”

“…We achieve this by modifying the source side of the parent training data, artificially changing its word orders with random noises (Figure 3). The noise function includes (Hill et al, 2016;Kim et al, 2018):…”

“…In phrase-based SMT, Alkhouli et al (2014) builds translation models with word/phrase embeddings. Kim et al (2018) uses cross-lingual word embedding as a basic translation model for unsupervised MT and attach other components on top of it. Artetxe et al (2018c) and Lample et al (2018a) initialize their unsupervised NMT models with pre-trained crosslingual word embeddings.…”

“…Artificial noises for the source sentences are used to counteract word-by-word training data in unsupervised MT (Artetxe et al, 2018c;Lample et al, 2018a;Kim et al, 2018), but in this work, they are used to regularize the NMT. Neubig and Hu (2018) study adapting a multilingual NMT system to a new language.…”

CUED@WMT19:EWC&LMs

“…Fine-tuning (Luong and Manning, 2015) is a domain adaptation technique that first trains a model 4 We use Sergey Edunov's addnoise.py script available at https://gist.github.com/edunov/ d67d09a38e75409b8408ed86489645dd until it converges on a training corpus A, and then continues training on a usually much smaller corpus B which is close to the target domain. Similarly to Schamper et al (2018); Koehn et al (2018a), we fine-tune our models on former WMT test sets (2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015)(2016) to adapt them to the target domain of high-quality news translations. Due to the very small size of corpus B, much care has to be taken to avoid over-fitting.…”