Translating Negation: A Manual Error Analysis

Abstract

Statistical Machine Translation has come a long way improving the translation quality of a range of different linguistic phenomena. With negation however, techniques proposed and implemented for improving translation performance on negation have simply followed from the developers' beliefs about why performance is worse. These beliefs, however, have never been validated by an error analysis of the translation output. In contrast, the current paper shows that an informative empirical error analysis can be formulated in terms of (1) the set of semantic elements involved in the meaning of negation, and (2) a small set of string-based operations that can characterise errors in the translation of those elements. Results on a Chinese-to-English translation task confirm the robustness of our analysis cross-linguistically and the basic assumptions can inform an automated investigation into the causes of translation errors. Conclusions drawn from this analysis should guide future work on improving the translation of negative sentences.

How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

“…85.2%, and 82.3%, respectively. Polarity shifts between the source and target text are a well-known translation problem, and our analysis shows that the main type of error is the deletion of negation markers, in line with with findings of previous studies (Fancellu and Webber, 2015). We consider the relatively high number of errors related to polarity an important problem in machine translation, and hope that future work will try to improve upon our results, shown in more detail in Table 5.…”

It’s not a Non-Issue: Negation as a Source of Error in Machine Translation

“…After sorting sentences according to Z-score, we compare reference translation and system output and annotate (Yes/No) whether the system gets negation wrong, compared to the reference. In contrast to Fancellu and Webber (2015), who do fine-grained annotation of translation errors related to negation (focusing on deletion/insertion/substitution of cues, focus, and scope), we ask a broader question designed to capture semantic adequacy focused on handling of negation. We only choose Yes if the system output shows a glaring error in meaning related to negation (see Section 2 for discussion of some typical error types).…”

“…Other linguistic phenomena and analysis axes have gathered significant attention in NMT evaluation studies, including anaphora resolution (Hardmeier et al, 2014;Voita et al, 2018) and pronoun translation (Guillou and Hardmeier, 2016), modality (Baker et al, 2012), ellipsis and deixis (Voita et al, 2019), word sense disambiguation (Tang et al, 2018), and morphological competence (Burlot and Yvon, 2017). Nevertheless, the last comprehensive study of the effect of negation in MT pertains to older, phrase-based models (Fancellu and Webber, 2015). In this work, we set out to study the effect of negation in modern NMT systems.…”

Recent advances in processing negation

“…Outside sentiment analysis, researchers have pointed out that negation poses unsolved challenges for, among others, machine translation and natural language inference. Fancellu and Webber (2015) present a manual error analysis translating negation from Chinese to English, and Bentivogli et al . (2016) point out that neural machine translation struggles as much as statistical machine translation when it comes to translating negation.…”