Metrics for Translation Quality Assessment: A Case for Standardising Error Typologies

Lommel, Arle

doi:10.1007/978-3-319-91241-7_6

Cited by 37 publications

(29 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These observations are closely related to the low interannotator agreement of error classification tasks (Lommel, Popović and Burchardt 2014; Popović 2018), which can be attributed to factors such as disagreement as to error spans, ambiguity between categories and disagreement as to the existence and the severity of errors (Lommel, Popović and Burchardt 2014; Popović 2018). As Lommel, Popović and Burchardt (2014) point out, the deficiencies reported in their work have allowed the improvement of the annotation guidelines; nonetheless, they cannot be expected to be thoroughly eradicated, as they are ‘inherent in the quality assessment task’.…”

Section: Human Evaluation Techniques Of Mt Qualitymentioning

confidence: 99%

“…The answer is: with strict criteria and evaluators’ training. Advance has been made to this direction, but still important issues, such as the number of reference translations, ratings and postedits required for a reliable evaluation, remains unclear (Lommel, Popović and Burchardt 2014). Last but not least, Läubli et al .…”

Section: Human Evaluation Techniques Of Mt Qualitymentioning

confidence: 99%

See 1 more Smart Citation

How to evaluate machine translation: A review of automated and human metrics

Chatzikoumi

2019

Nat. Lang. Eng.

View full text Add to dashboard Cite

This article presents the most up-to-date, influential automated, semiautomated and human metrics used to evaluate the quality of machine translation (MT) output and provides the necessary background for MT evaluation projects. Evaluation is, as repeatedly admitted, highly relevant for the improvement of MT. This article is divided into three parts: the first one is dedicated to automated metrics; the second, to human metrics; and the last, to the challenges posed by neural machine translation (NMT) regarding its evaluation. The first part includes reference translation–based metrics; confidence or quality estimation (QE) metrics, which are used as alternatives for quality assessment; and diagnostic evaluation based on linguistic checkpoints. Human evaluation metrics are classified according to the criterion of whether human judges directly express a so-called subjective evaluation judgment, such as ‘good’ or ‘better than’, or not, as is the case in error classification. The former methods are based on directly expressed judgment (DEJ); therefore, they are called ‘DEJ-based evaluation methods’, while the latter are called ‘non-DEJ-based evaluation methods’. In the DEJ-based evaluation section, tasks such as fluency and adequacy annotation, ranking and direct assessment (DA) are presented, whereas in the non-DEJ-based evaluation section, tasks such as error classification and postediting are detailed, with definitions and guidelines, thus rendering this article a useful guide for evaluation projects. Following the detailed presentation of the previously mentioned metrics, the specificities of NMT are set forth along with suggestions for its evaluation, according to the latest studies. As human translators are the most adequate judges of the quality of a translation, emphasis is placed on the human metrics seen from a translator-judge perspective to provide useful methodology tools for interdisciplinary research groups that evaluate MT systems.

show abstract

Section: Human Evaluation Techniques Of Mt Qualitymentioning

confidence: 99%

Section: Human Evaluation Techniques Of Mt Qualitymentioning

confidence: 99%

How to evaluate machine translation: A review of automated and human metrics

Chatzikoumi

2019

Nat. Lang. Eng.

View full text Add to dashboard Cite

show abstract

“…How accuracy and quality are measured varies from one TC to another. For an overview of the evolution of Translation Quality Assessment (TQA) in the profession and the current state of play, including details of the Multidimensional Quality Metrics (MQM), see Lommel (2018).…”

Section: Resultsmentioning

confidence: 99%

Beyond the margins of academic education: identifying translation industry training practices through action research

Cespedes¹

2020

Trans-Int

View full text Add to dashboard Cite

show abstract

“…There has traditionally been, as in the case of that which occurs with quality assessment, a lack of standardisation for MT error analysis and classification (Lommel, 2018). In this regard, many authors (Costa-Jussà & Farrús, 2015;Farrús et al, 2010;Krings, 2001;Laurian, 1984;Schäfer, 2003;Vilar et al, 2006) have proposed different typologies and classifications for errors related to MT and, generally speaking, the majority of these distinguish errors at different levels (spelling, vocabulary, grammar, discourse), divided into subcategories that include, for example, errors relating to concordance, style, confusion in word meaning with various exceptions, etc.…”

Section: Classification and Analysis Of Mt Errorsmentioning

confidence: 99%

Machine translation systems and quality assessment: a systematic review

Rivera-Trigueros

2021

Lang Resources & Evaluation

View full text Add to dashboard Cite

Nowadays, in the globalised context in which we find ourselves, language barriers can still be an obstacle to accessing information. On occasions, it is impossible to satisfy the demand for translation by relying only in human translators, therefore, tools such as Machine Translation (MT) are gaining popularity due to their potential to overcome this problem. Consequently, research in this field is constantly growing and new MT paradigms are emerging. In this paper, a systematic literature review has been carried out in order to identify what MT systems are currently most employed, their architecture, the quality assessment procedures applied to determine how they work, and which of these systems offer the best results. The study is focused on the specialised literature produced by translation experts, linguists, and specialists in related fields that include the English–Spanish language combination. Research findings show that neural MT is the predominant paradigm in the current MT scenario, being Google Translator the most used system. Moreover, most of the analysed works used one type of evaluation—either automatic or human—to assess machine translation and only 22% of the works combined these two types of evaluation. However, more than a half of the works included error classification and analysis, an essential aspect for identifying flaws and improving the performance of MT systems.

show abstract

Metrics for Translation Quality Assessment: A Case for Standardising Error Typologies

Cited by 37 publications

References 0 publications

How to evaluate machine translation: A review of automated and human metrics

How to evaluate machine translation: A review of automated and human metrics

Beyond the margins of academic education: identifying translation industry training practices through action research

Machine translation systems and quality assessment: a systematic review

Contact Info

Product

Resources

About