A Pronoun Test Suite Evaluation of the English–German MT Systems at WMT 2018

Abstract

We evaluate the output of 16 English-to-German MT systems with respect to the translation of pronouns in the context of the WMT 2018 competition. We work with a test suite specifically designed to assess system quality in various fine-grained categories known to be problematic. The main evaluation scores come from a semi-automatic process, combining automatic reference matching with extensive manual annotation of uncertain cases. We find that current NMT systems are good at translating pronouns with intra-sentential reference, but the inter-sentential cases remain difficult. NMT systems are also good at the translation of event pronouns, unlike systems from the phrase-based SMT paradigm. No single system performs best at translating all types of anaphoric pronouns, suggesting unexplained random effects influencing the translation of pronouns with NMT.

Analysing concatenation approaches to document-level NMT in two different domains

“…The PROTEST evaluation confirms the findings of the WMT18 evaluation (Guillou et al, 2018). In both of these evaluations the pleonastic and event categories are the least problematic.…”

“…Throughout, however, particular discourse phenomena are consistently targeted, as they are indeed indicators of globally good, cohesive and coherent texts. Pronouns (Hardmeier and Federico, 2010;Guillou, 2012;Hardmeier et al, 2013;Guillou et al, 2018) have been largely at the center of attention, and more recently the translation of pronouns in the context of their coreferential chains has been looked at (Lapshinova-Koltunski and Hardmeier, 2017;Lapshinova-Koltunski et al, 2019). Other devices studied are verbal tenses (Gong et al, 2012;Loáiciga et al, 2014;Ramm and Fraser, 2016) and connectives , although not using neural models.…”

“…At the differences are not significant. least, we are able to evaluate the quality of pronoun translation thanks to the existence of two test suites for English-German pronoun translation: PROTEST Guillou et al, 2018) is based on TED talks transcripts. These consist of planned speech documents, therefore the genre is somewhere in the middle between news text and dialog.…”

When and Why is Document-level Context Useful in Neural Machine Translation?

“…Recently, it has been claimed that sentence-level NMT generates document-level errors, e.g. wrong coreference of pronouns/articles or inconsistent translations throughout a document (Guillou et al, 2018;Läubli et al, 2018).…”

The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Translation

“…Existing test suites focus e.g. on morphosyntactic and syn-tactic divergences between source and target language (Burchardt et al, 2017;Burlot and Yvon, 2017;Isabelle et al, 2017;Sennrich, 2017;Burlot et al, 2018;Macketanz et al, 2018) or on discourse phenomena (Guillou and Hardmeier, 2016;Bawden et al, 2018;Guillou et al, 2018).…”