Controlled Translation in an Example-based Environment: What do Automatic Evaluation Metrics Tell Us?

Abstract

This paper presents an extended, harmonised account of our previous work on integrating controlled language data in an Example-based Machine Translation system. Gough and Way in MT Summit pp. 133-140 (2003) focused on controlling the output text in a novel manner, while Gough and Way (9th Workshop of the EAMT, (2004a), pp. 73-81) sought to constrain the input strings according to controlled language specifications. Our original sub-sentential alignment algorithm could deal only with 1:1 matches, but subsequent refinements enabled n:m alignments to be captured. A direct consequence was that we were able to populate the system's databases with more than six times as many potentially useful fragments. Together with two simple novel improvements -correcting a small number of mistranslations in the lexicon, and allowing multiple translations in the lexicon -translation quality improves considerably. We provide detailed automatic and human evaluations of a number of experiments carried out to test the quality of the system. We observe that our system outperforms the rule-based on-line system Logomedia on a range of automatic evaluation metrics, and that the 'best' translation candidate is consistently highly ranked by our system. Finally, we note in a number of tests that the BLEU metric gives objectively different results than other automatic evaluation metrics and a manual evaluation. Despite these conflicting results, we observe a preference for controlling the source data rather than the target translations.

Hybrid data-driven models of machine translation

“…Rather than using models of syntax in a post hoc fashion, as is the case with most SMT systems, most EBMT models of translation build in syntax a t their core, during both the creation of their knowledge resources and during the translation process itself. IIowever, EBMT systems st111 tend to suffer from problems of coverage and seem to perform better in controlled language (Way and Gough, 2005b) or sublanguage domains (Sumita et al, 1990).…”