Using Multiple Edit Distances to Automatically Grade Outputs From Machine Translation Systems

Abstract

This paper addresses the challenging problem of automatically evaluating output from machine translation (MT) systems in order to support the developers of these systems. Conventional approaches to the problem include methods that automatically assign a rank such as A, B, C, or D to MT output according to a single edit distance between this output and a correct translation example. The single edit distance can be differently designed, but changing its design makes assigning a certain rank more accurate, but another rank less accurate. This inhibits improving accuracy of rank assignment. To overcome this obstacle, this paper proposes an automatic ranking method that, by using multiple edit distances, encodes machine-translated sentences with a rank assigned by humans into multi-dimensional vectors from which a classifier of ranks is learned in the form of a decision tree (DT). The proposed method assigns a rank to MT output through the learned DT. The proposed method is evaluated using transcribed texts of real conversations in the travel arrangement domain. Experimental results show that the proposed method is more accurate than the single-edit-distance-based ranking methods, in both closed and open tests. Moreover, the proposed method could estimate MT quality within 3% error in some cases.

Assessing the Translation of Google and Microsoft Bing in Translating Political Texts from Arabic into English

“…The evaluation of MT on Arabic to English and Mandarin to English ranks MT output in comparison to referenced human translation by an expert to judge which output is closer to human translation. Reference [7] in Using Multiple Edit Distances to Automatically Grade Outputs from Machine Translation Systems presented an evaluation method that is a subsystem of SSMT (Speech-to-speech MT systems. The method is "Grader based on Edit Distance" that compute the score of MT output by using a decision tree.…”

Evaluating English to Arabic Machine Translation Using BLEU

“…Another method for evaluating machine translation systems was presented by Akiba et al [11]. Their study was dedicated to evaluate machine translation (MT) systems that are subsystems of speech-to-speech MT (SSMT) systems.…”

Predicting Human Assessment of Machine Translation Quality by Combining Automatic Evaluation Metrics using Binary Classifiers

“…[Quirk 1994] also investigates the feasibility of various learning approaches for the multiclass classification problem for a very small data set in the domain of technical documentation. [Akiba et al 2001] [2] utilized DT classifiers trained on multiple edit-distance features where combinations of lexical (stem, word, part-of-speech) and semantic (thesaurus-based semantic class) matches were used to compare MT system outputs with reference translations and to approximate human scores of acceptability directly. [Kulesza and Shieber 2004] [13] trained a binary SVM classifier based on automatic scoring features in order to distinguish between "human-produced" and "machinegenerated" translations of newswire data instead of predicting human judgments directly.…”