Applications of automatic evaluation methods to measuring a capability of speech translation system

Abstract

The main goal of this paper is to propose automatic schemes for the translation paired comparison method. This method was proposed to precisely evaluate a speech translation system's capability. Furthermore, the method gives an objective evaluation result, i.e., a score of the Test of English for International Communication (TOEIC). The TOEIC score is used as a measure of one's speech translation capability. However, this method requires tremendous evaluation costs. Accordingly, automatization of this method is an important subject for study. In the proposed method, currently available automatic evaluation methods are applied to automate the translation paired comparison method.In the experiments, several automatic evaluation methods (BLEU, NIST, DPbased method) are applied. The exper-imental results of these automatic measures show a good correlation with evaluation results of the translation paired comparison method.

An automatic method for summary evaluation using multiple evaluation results by a manual method

“…Yasuda et al (2003) tested DP matching (Su et al, 1992), BLEU (Papineni et al, 2002) (Yasuda et al, 2003) is another method of combining several manual scores.…”

“…Yasuda et al (2003) tested DP matching (Su et al, 1992), BLEU (Papineni et al, 2002), and NIST 3 , as automatic methods used in their evaluation. Instead of those methods, we tested ROUGE and cosine distance, both of which have been used for summary evaluation.…”

“…Yasuda et al (2003) proposed an automatic method that gives an evaluation result of a translation system as a score for TOEIC. They prepared 29 human subjects, whose TOEIC scores were from 300s to 800s, and asked them to translate 23 Japanese conversations into English.…”

“…There are two research studies related to our work (Kazawa et al, 2003, Yasuda et al, 2003. Kazawa et al (2003) proposed an automatic evaluation method using multiple evaluation results from a manual method.…”

“…Kazawa et al (2003) proposed an automatic evaluation method using multiple evaluation results from a manual method. In the field of machine translation, Yasuda et al (2003) proposed an automatic method that gives an evaluation result of a translation system as a score for the Test of English for International Communication (TOEIC). Although the effectiveness of both methods was confirmed experimentally, further discussion of four points, which we describe in Section 3, is necessary for a more accurate summary evaluation.…”