Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics

Lommel, Arle; Uszkoreit, Hans; Burchardt, Aljoscha

doi:10.5565/rev/tradumatica.77

Cited by 131 publications

(124 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this paper we conduct a detailed human analysis of the outputs produced by NMT and PBMT systems. Namely, we annotate manually the errors found according to a detailed error taxonomy, that is compliant with the hierarchical listing of issue types defined as part of the Multidimensional Quality Metrics (MQM) (Lommel et al, 2014a). Specifically, we carry out this analysis for the news domain in the English-to-Croatian language direction.…”

Section: June 2017mentioning

confidence: 99%

Fine-Grained Human Evaluation of Neural Versus Phrase-Based Machine Translation

Klubička

Toral

Sánchez-Cartagena³

2017

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

We compare three approaches to statistical machine translation (pure phrase-based, factored phrase-based and neural) by performing a fine-grained manual evaluation via error annotation of the systems' outputs. The error types in our annotation are compliant with the multidimensional quality metrics (MQM), and the annotation is performed by two annotators. Inter-annotator agreement is high for such a task, and results show that the best performing system (neural) reduces the errors produced by the worst system (phrase-based) by 54%.

show abstract

Section: June 2017mentioning

confidence: 99%

Fine-Grained Human Evaluation of Neural Versus Phrase-Based Machine Translation

Klubička

Toral

Sánchez-Cartagena³

2017

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

show abstract

“…We manually annotate the errors found according to a detailed error taxonomy that is compliant with the hierarchical listing of issue types defined as part of the Multidimensional Quality Metrics (MQM) (Lommel et al, 2014a). First, we define an error taxonomy that is relevant to the problematic linguistic phenomena of this language pair.…”

Section: Introductionmentioning

confidence: 99%

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Klubička¹,

Toral

Sánchez-Cartagena³

2018

Machine Translation

View full text Add to dashboard Cite

This paper presents a quantitative fine-grained manual evaluation approach to comparing the performance of different machine translation (MT) systems. We build upon the well-established Multidimensional Quality Metrics (MQM) error taxonomy and implement a novel method that assesses whether the differences in performance for MQM error types between different MT systems are statistically significant. We conduct a case study for English-toCroatian, a language direction that involves translating into a morphologically rich language, for which we compare three MT systems belonging to different paradigms: pure phrase-based, factored phrase-based and neural. First, we design an MQM-compliant error taxonomy tailored to the relevant linguistic phenomena of Slavic languages, which made the annotation process feasible and accurate. Errors in MT outputs were then annotated by two annotators following this taxonomy. Subsequently, we carried out a statistical analysis which showed that the best-performing system (neural) reduces the errors produced by the worst system (pure phrase-based) by more than half (54%). Moreover, we conducted an additional analysis of agreement errors in which we distinguished between short (phrase-level) and long distance (sentence-level) errors. We discovered that phrase-based MT approaches are of limited use for long distance agreement phenomena, for which neural MT was found to be especially effective.

show abstract

“…In production environments, for example, flexible frameworks like MQM-DQF (Lommel, Uszkoreit and Burchardt, 2014;qt21, 2015) are often preferred and are used in the evaluation of both human and machine translation, while in MT labs, reference-based automatic metrics-of which there is a multitude-have tended to prevail in the evaluation of machine output. (González, Giménez and Girona Salgado (2014), for example, list some 60 automatic evaluation metrics, or variations thereof, based on similarity with a reference translation).…”

Section: Quality Metricsmentioning

confidence: 99%