How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

Abstract

Analysing translation quality in regards to specific linguistic phenomena has historically been difficult and time-consuming. Neural machine translation has the attractive property that it can produce scores for arbitrary translations, and we propose a novel method to assess how well NMT systems model specific linguistic phenomena such as agreement over long distances, the production of novel words, and the faithful translation of polarity. The core idea is that we measure whether a reference translation is more probable under a NMT model than a contrastive translation which introduces a specific type of error. We present LingEval97 1 , a large-scale data set of 97 000 contrastive translation pairs based on the WMT English→German translation task, with errors automatically created with simple rules. We report results for a number of systems, and find that recently introduced character-level NMT systems perform better at transliteration than models with byte-pair encoding (BPE) segmentation, but perform more poorly at morphosyntactic agreement, and translating discontiguous units of meaning.

An Analysis of Source-Side Grammatical Errors in NMT

“…Our goal instead, is to analyze the behavior of the MT system when confronted with ungrammatical input. Reference-less evaluation has also been proposed for text simplification (Martin et al, 2018) and GEC (Napoles et al, 2016), while the grammaticality of MT systems' outputs has been evaluated with target-side contrastive pairs (Sennrich, 2017). In this work, the core of our evaluation of a system's robustness lies in the following observation: a perfectly robust-to-noise MT system would produce the exact same output for the clean and erroneous versions of the same input sentence.…”

Deep architectures for Neural Machine Translation

“…Here, we seek to answer this question by testing our models on Lingeval97 (Sennrich, 2017), a test set which provides contrastive translation pairs for different types of errors. For the example of subject-verb agreement, contrastive translations are created from a reference translation by changing the grammatical number of the verb, and we can measure how often the NMT model prefers the correct reference over the contrastive variant.…”

Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation

“…Unsupervised NMT The current NMT systems (Sutskever et al, 2014;Cho et al, 2014a;Bahdanau et al, 2015;Gehring et al, 2017;Vaswani et al, 2017) are known to easily overfit and result in an inferior performance when the training data is limited (Koehn and Knowles, 2017;Isabelle et al, 2017;Sennrich, 2017). Many research efforts have been spent on how to utilize the monolingual data to improve the NMT system when only limited supervision is available (Gulcehre et al, 2015;Sennrich et al, 2016a;He et al, 2016;Zhang and Zong, 2016;.…”