Towards Automatic Error Analysis of Machine Translation Output

Abstract

Evaluation and error analysis of machine translation output are important but difficult tasks. In this article, we propose a framework for automatic error analysis and classification based on the identification of actual erroneous words using the algorithms for computation of Word Error Rate (WER) and Position independent word Error Rate (PER) which is just a very first step towards development of automatic evaluation measures which provide more specific information of certain translation problems. The proposed approach enables the use of various types of linguistic knowledge in order to classify translation errors in many different ways. This work focuses on one possible set-up, namely on five error categories: inflectional errors, errors due to wrong word order, missing words, extra words and incorrect lexical choices. For each of the categories, we analyse the contribution of various POS classes. We compared the results of automatic error analysis with the results of human error analysis in order to investigate two possible applications: estimating the contribution of each error type in a given translation output in order to identify the main sources of errors for a given translation system, and comparing different translation outputs using the introduced error categories in order to obtain more information about advantages and disadvantages of different systems and possibilites for improvements, as well as about advantages and disadvantages of applied methods for improvements. We used Arabic-English Newswire and Broadcast News and Chinese-English Newswire outputs created in the framework of the GALE project, several Spanish and English European Parliament outputs generated during the TC-Star project, and three German-English outputs generated in the framework of the fourth Machine Translation Workshop (WMT). We show that our results correlate very well with the results of a human error analysis, and that all our metrics except the extra words reflect well the differences between different versions of the same translation system as well as the differences between different translation systems.

Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output

“…It implements the method based on the standard word error rate () combined with the precision and recall based error rates (Popović and Ney, 2007) and it has been tested on various language pairs and tasks. It is shown that the obtained results have high correlation (between 0.6 and 1.0) with the results obtained by human evaluators (Popović and Burchardt, 2011;Popović and Ney, 2011).…”

A linguistically motivated taxonomy for Machine Translation error analysis

“…Furthermore, in Fishel et al (2012) a collection of annotated translation errors is presented, consisting of automatically produced translations and their detailed manual analysis. 14 Using the collected corpora, the authors evaluated two available state-of-the-art methods of MT diagnostics and assessment: Addicter (Zeman et al 2011) 15 and Hjerson (Popović and Ney 2011). 16 Addicter is an open-source tool that uses a method explicitly based on aligning the hypothesis and reference translations to devise the various error types.…”

Assessing the Impact of Translation Errors on Machine Translation Quality with Mixed-effects Models

“…So far, the relation between errors and automatic metrics has been analysed by measuring the correlation between single or total error frequencies and automatic scores (Popović and Ney, 2011;Farrús et al, 2012). Using two different error taxonomies, both works show that the sum of the errors has a high correlation with BLEU and TER scores.…”