ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation

Abstract

Recent high scores on pronoun translation using context-aware neural machine translation have suggested that current approaches work well. ContraPro is a notable example of a contrastive challenge set for English→German pronoun translation. The high scores achieved by transformer models may suggest that they are able to effectively model the complicated set of inferences required to carry out pronoun translation. This entails the ability to determine which entities could be referred to, identify which entity a sourcelanguage pronoun refers to (if any), and access the target-language grammatical gender for that entity. We first show through a series of targeted adversarial attacks that in fact current approaches are not able to model all of this information well. Inserting small amounts of distracting information is enough to strongly reduce scores, which should not be the case. We then create a new template test set Contracat, designed to individually assess the ability to handle the specific steps necessary for successful pronoun translation. Our analyses show that current approaches to context-aware nmt rely on a set of surface heuristics, which break down when translations require real reasoning. We also propose an approach for augmenting the training data, with some improvements.

Measuring and Increasing Context Usage in Context-Aware Machine Translation

“…Müller et al (2018) built a large-scale dataset for anaphoric pronoun resolution, Bawden et al (2018) manually created a dataset for both pronoun resolution and lexical choice and Voita et al (2019) created a dataset that targets deixis, ellipsis and lexical cohesion. Stojanovski et al (2020) showed through adversarial attacks that models that do well on other contrastive datasets rely on surface heuristics and create a contrastive dataset to address this. In contrast, our CXMI metric is phenomenon-agnostic and can be measured with respect to all phenomena that require context in translation.…”

Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution

“…Such undesirable behaviour is facilitated by dataset biases that models are exposed to during training (Emelin et al, 2020). In their study of coreference, (Stojanovski et al, 2020) indicate that gender and positional biases can influence model behavior. To verify whether this is the case for cross-lingual Winograd schemas, we examine how strongly pronoun gender and the relative antecedent position correlates with model preference.…”

“…Similarly, the study of coreference has a long tradition in machine translation. Several CoR datasets have been proposed in the past, including (Guillou and Hardmeier, 2016;Bawden et al, 2018;Müller et al, 2018;Stojanovski et al, 2020). Among those, that of (Stojanovski et al, 2020) is most relevant to our work.…”

“…Several CoR datasets have been proposed in the past, including (Guillou and Hardmeier, 2016;Bawden et al, 2018;Müller et al, 2018;Stojanovski et al, 2020). Among those, that of (Stojanovski et al, 2020) is most relevant to our work. While it contains samples that require world knowledge to resolve coreference, they are constructed from a fixed set of templates and remain limited to EN-DE.…”

On the Limits of Minimal Pairs in Contrastive Evaluation

“…in a sequence-to-sequence task, human-written pairs (or pairs that are machinegenerated to be different from the training distribution on purpose) may tell us more about the robustness of models outside the mode. For example, terminology-constrained or interactive applications depend on robustness against improbable contexts, and contrastive evaluation indicates that current NMT systems lack such robustness (Stojanovski et al, 2020). Similarly, syntactic evaluation of language models using randomly generated or nonsensical sentences (Gulordava et al, 2018;Warstadt et al, 2020) can be seen as method to assess the robustness of a model under improbable input, rather than as an assessment of generative capabilities in general.…”