Morpho-syntactic information for automatic error analysis of statistical machine translation output

Popović, Maja; Gispert, Adrià de; Gupta, Deepa; Lambert, Patrik; Ney, Hermann; Acebal, José Bernardo Mariño; Federico, Marcello; Banchs, Rafael E.

doi:10.3115/1654650.1654652

Cited by 22 publications

(14 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A method for automatic identification of patterns in translation output using POS sequences is proposed by Lopez and Resnik (2005) in order to see how well a translation system is capable of capturing systematic reordering patterns. Using relative differences between WER and PER for three POS classes (nouns, adjectives and verbs) is proposed by Popović et al (2006) for the estimation of inflectional and reordering errors. Semi-automatic error analysis (Kirchhoff et al 2007) is carried out in order to identify problematic characteristics of source documents such as genre, domain, language, etc.…”

Section: Introductionmentioning

confidence: 99%

Towards Automatic Error Analysis of Machine Translation Output

Popović

Ney

2011

Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Evaluation and error analysis of machine translation output are important but difficult tasks. In this article, we propose a framework for automatic error analysis and classification based on the identification of actual erroneous words using the algorithms for computation of Word Error Rate (WER) and Position independent word Error Rate (PER) which is just a very first step towards development of automatic evaluation measures which provide more specific information of certain translation problems. The proposed approach enables the use of various types of linguistic knowledge in order to classify translation errors in many different ways. This work focuses on one possible set-up, namely on five error categories: inflectional errors, errors due to wrong word order, missing words, extra words and incorrect lexical choices. For each of the categories, we analyse the contribution of various POS classes. We compared the results of automatic error analysis with the results of human error analysis in order to investigate two possible applications: estimating the contribution of each error type in a given translation output in order to identify the main sources of errors for a given translation system, and comparing different translation outputs using the introduced error categories in order to obtain more information about advantages and disadvantages of different systems and possibilites for improvements, as well as about advantages and disadvantages of applied methods for improvements. We used Arabic-English Newswire and Broadcast News and Chinese-English Newswire outputs created in the framework of the GALE project, several Spanish and English European Parliament outputs generated during the TC-Star project, and three German-English outputs generated in the framework of the fourth Machine Translation Workshop (WMT). We show that our results correlate very well with the results of a human error analysis, and that all our metrics except the extra words reflect well the differences between different versions of the same translation system as well as the differences between different translation systems.

show abstract

Section: Introductionmentioning

confidence: 99%

Towards Automatic Error Analysis of Machine Translation Output

Popović

Ney

2011

Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Automatic classifications. Shortly after the classification made by Vilar et al [2006], the importance of using linguistic information was acknowledged by [Popović et al 2006], together with the need of automatizing the process, considering that a human error analysis and error classification was a very time consuming task. Therefore, the authors propose the use of morpho-syntactic information in combination with Giménez and Màrquez 2007] syntaxis, independent, automatic metrics shallow-semantics not limited to lexis syntactic structure automatic, language-pair [Popović and Ney 2009] of the sentence independent, based on BLEU, TER and METEOR metrics orthography, morphology, manual, language-pair [Farrús et al 2010] lexis, semantics, syntaxis dependent (Catalan-Spanish), generalizable to any lang.…”

Section: Language-pair Dependent Linguistic Evaluation Methodsmentioning

confidence: 99%

Statistical machine translation enhancements through linguistic levels

Costa-jussà

Farrús

2014

ACM Comput. Surv.

View full text Add to dashboard Cite

Machine translation can be considered a highly interdisciplinary and multidisciplinary field because it is approached from the point of view of human translators, engineers, computer scientists, mathematicians and linguists. One of the most popular approaches is the statistical machine translation (SMT) approach which tries to cover translation in a holistic manner by learning from parallel corpus aligned at the sentence level. However, with this basic approach, there are some issues at each written linguistic level (i.e. orthographic, morphological, lexical, syntactic and semantic) that remain unsolved. Research in SMT has continuously been focused on solving the different linguistic levels challenges. This paper represents a survey of how the SMT has been enhanced to perform translation correctly at all linguistic levels.

show abstract

“…Error analysis techniques have not been substantially explored, although it has recently been identified as an important task [Och 2005]. A few techniques for error analysis [DeNeefe et al 2005;Chiang et al 2005;Popovic et al 2006] and confidence estimation [Ueffing and Ney 2005] have been investigated, but in general this area remains underexplored.…”

Section: Current Directions and Future Researchmentioning

confidence: 99%

Statistical machine translation

2008

View full text Add to dashboard Cite

Statistical machine translation (SMT) treats the translation of natural language as a machine learning problem. By examining many samples of human-produced translation, SMT algorithms automatically learn how to translate. SMT has made tremendous strides in less than two decades, and new ideas are constantly introduced. This survey presents a tutorial overview of the state of the art. We describe the context of the current research and then move to a formal problem description and an overview of the main subproblems: translation modeling, parameter estimation, and decoding. Along the way, we present a taxonomy of some different approaches within these areas. We conclude with an overview of evaluation and a discussion of future directions.

show abstract

Morpho-syntactic information for automatic error analysis of statistical machine translation output

Cited by 22 publications

References 10 publications

Towards Automatic Error Analysis of Machine Translation Output

Towards Automatic Error Analysis of Machine Translation Output

Statistical machine translation enhancements through linguistic levels

Statistical machine translation

Contact Info

Product

Resources

About