Evaluating Conjunction Disambiguation on English-to-German and French-to-German WMT 2019 Translation Hypotheses

Abstract

We present a test set for evaluating an MT system's capability to translate ambiguous conjunctions depending on the sentence structure. We concentrate on the English conjunction "but" and its French equivalent "mais" which can be translated into two different German conjunctions. We evaluate all English-to-German and French-to-German submissions to the WMT 2019 shared translation task. The evaluation is done mainly automatically, with additional fast manual inspection of unclear cases. All systems almost perfectly recognise the target conjunction "aber", whereas accuracies for the other target conjunction "sondern" range from 78% to 97%, and the errors are mostly caused by replacing it with the alternative conjunction "aber". The best performing system for both language pairs is a multilingual Transformer TartuNLP system trained on all WMT 2019 language pairs which use the Latin script, indicating that the multilingual approach is beneficial for conjunction disambiguation. As for other system features, such as using synthetic back-translated data, context-aware, hybrid, etc., no particular (dis)advantages can be observed. Qualitative manual inspection of translation hypotheses shown that highly ranked systems generally produce translations with high adequacy and fluency, meaning that these systems are not only capable of capturing the right conjunction whereas the rest of the translation hypothesis is poor. On the other hand, the low ranked systems generally exhibit lower fluency and poor adequacy.

Findings of the 2019 Conference on Machine Translation (WMT19)

“…English and French (Popović, 2019) The test suite by Popović ( 2019) contains approximately 1000 English and 1000 individual French sentences that were included in the English→German and French→German tasks. The sentences focus on the translation of the English "but" and French "mais" which should be disambiguated into German "aber" or "sondern".…”

A Comparison between Pre-training and Large-scale Back-translation for Neural Machine Translation

“…Conjunction can impact the structure of the surrounding sentences and is related more to fluency than to adequacy. Therefore it can be more difficult than content word ambiguity (Popović, 2019). We conclude that BERT can actually absorb fine-grained relevant sense information during pre-training, which helps learn meaningful conjunction sense distinctions.…”

A Survey on Document-level Neural Machine Translation: Methods and Evaluation

“…The test suites introduced by [2], [86] and [85] were also used to evaluate document-level MT systems at WMT19 but these consist of sentence-pairs only and do not rely on any extra-sentential context information. We conclude that an evaluation using test suites is feasible but has a restricted scope since it is designed for specific language-pairs and has limited guarantees.…”