Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output

Abstract

We describe Hjerson, a tool for automatic classification of errors in machine translation output. The tool features the detection of five word level error classes: morphological errors, reodering errors, missing words, extra words and lexical errors. As input, the tool requires original full form reference translation(s) and hypothesis along with their corresponding base forms. It is also possible to use additional information on the word level (e.g.  tags) in order to obtain more details. The tool provides the raw count and the normalised score (error rate) for each error class at the document level and at the sentence level, as well as original reference and hypothesis words labelled with the corresponding error class in text and  formats.

A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions

“…In this experiment we assess the performance of NMT versus PBMT systems on a set of error categories that correspond to five word-level error classes: inflection errors, reordering errors, missing words, extra words and incorrect lexical choices. These errors are detected automatically using the edit distance, word error rate (WER), precision-based and recall-based position- independent error rates (hPER and rPER, respectively) as implemented in Hjerson (Popović, 2011). These error classes are then defined as follows:…”

Comparing Machine Translation and Human Translation: A Case Study

“…A taxonomy which has been popular in MT is (Vilar et al, 2006). To avoid the necessity of calling in human evaluators every time an error analysis is to be performed there have also been work on automatic error classification (Popović and Burchardt, 2011). While simply counting errors seems less relevant for comparing machine translation to human translation, showing what type of errors occur can be useful.…”

A Linguistic Evaluation of Rule-Based, Phrase-Based, and Neural MT Engines

“…Due to the variation of language, ambiguity, etc., checking and evaluating MT output can be almost as difficult as the translation itself. Still, people have tried to automatically classify errors comparing MT output to reference translations or post-edited MT output using tools like Hjerson (Popovic, 2011).…”