Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Abstract

This paper presents a quantitative fine-grained manual evaluation approach to comparing the performance of different machine translation (MT) systems. We build upon the well-established Multidimensional Quality Metrics (MQM) error taxonomy and implement a novel method that assesses whether the differences in performance for MQM error types between different MT systems are statistically significant. We conduct a case study for English-toCroatian, a language direction that involves translating into a morphologically rich language, for which we compare three MT systems belonging to different paradigms: pure phrase-based, factored phrase-based and neural. First, we design an MQM-compliant error taxonomy tailored to the relevant linguistic phenomena of Slavic languages, which made the annotation process feasible and accurate. Errors in MT outputs were then annotated by two annotators following this taxonomy. Subsequently, we carried out a statistical analysis which showed that the best-performing system (neural) reduces the errors produced by the worst system (pure phrase-based) by more than half (54%). Moreover, we conducted an additional analysis of agreement errors in which we distinguished between short (phrase-level) and long distance (sentence-level) errors. We discovered that phrase-based MT approaches are of limited use for long distance agreement phenomena, for which neural MT was found to be especially effective.

Translation Quality and Error Recognition in Professional Neural Machine Translation Post-Editing

“…Previous studies on NMT quality reported omissions and mistranslations/incorrect lexis as problematic NMT error categories, e.g. [13,[17][18][19], and a similar picture is painted in our analysis of Figure 19. Interaction between log-transformed first fixation durations and Error Type for eye-key span (the time between first visual contact with a target word and the first keystroke that contributes to the correction of this token).…”

“…Previous studies on NMT quality reported omissions and mistranslations/incorrect lexis as problematic NMT error categories, e.g., [13,[17][18][19], and a similar picture is painted in our analysis of the DGT corpus, where lexical errors, and particularly stylistic/register errors, function words, mistranslations, and terminology errors are the most common error types-both in the machine translation output and the post-edits, which might be explained by the priming effect of machine translations. The most prominent error categories, i.e., mistranslations, terminology errors, function words and stylistic/register errors, were further analysed in a key-logging and eye-tracking experiment to gain more insights regarding their effect on PE effort indicators such as the eye-tracking measures.…”

“…Lacruz [16], e.g., also provided evidence that not only the number of errors but also the error type plays an important role in determining the actual PE-effort involved, because some errors seem to be more demanding than others. Recent studies observed that NMT reduces some error types more than others e.g., [17][18][19], some of them using process measures such as keystrokes or pauses. However, none of them has so far applied eye-tracking measures such as average fixation counts and durations, which, according to [20], correlate strongly with technical and temporal post-editing effort.…”

A user-study on online adaptation of neural machine translation to human post-edits

“…The advantages of NMT and its challenges have been investigated from different angles in recent work (Koehn and Knowles, 2017;Toral and Sánchez-Cartagena, 2017;Farajian et al, 2017;Macketanz et al, 2017;Castilho et al, 2017a;Klubička et al, 2017;Bentivogli et al, 2018;Isabelle et al, 2017;Popović, 2017;Forcada, 2017;Castilho et al, 2017b;Junczys-Dowmunt et al, 2016;Klubička et al, 2018;Shterionov et al, 2017;Burchardt et al, 2017, inter alia), however, studies on interactive NMT, especially user studies involving human post-edits of NMT outputs, have so far not been presented.…”

Evaluating Conjunction Disambiguation on English-to-German and French-to-German WMT 2019 Translation Hypotheses

“…Ambiguous words are often difficult to translate automatically, even by the current state-of-the-art neural machine (NMT) systems. Whereas NMT systems produce more fluent (grammatical and natural) translations than the previous state-of-theart statistical phrase-based (PBMT) models, the semantic faithfulness of the translation to the original (adequacy) is still often problematic (Castilho et al, 2017;Klubička et al, 2018). Adequacy is even more problematic for ambiguous words which have two or more meanings depending on the context.…”