Using a mixture of N-best lists from multiple MT systems in rank-sum-based confidence measure for MT outputs

Abstract

This paper addressees the problem of eliminating unsatisfactory outputs from machine translation (MT) systems. The authors intend to eliminate unsatisfactory MT outputs by using confidence measures. Confidence measures for MT outputs include the rank-sum-based confidence measure (RSCM) for statistical machine translation (SMT) systems. RSCM can be applied to non-SMT systems but does not always work well on them. This paper proposes an alternative RSCM that adopts a mixture of the N-best lists from multiple MT systems instead of a single-system's N-best list in the existing RSCM. In most cases, the proposed RSCM proved to work better than the existing RSCM on two non-SMT systems and to work as well as the existing RSCM on an SMT system. * This research was supported in part by the Ministry of Public Management, Home Affairs, Posts and Telecommunications, Japan. 1 These confidence measures are a kind of automatic evaluator such as mWER (Niessen et al., 2000) and BLEU (Papineni et al., 2001). While mWER and BLEU cannot be used online, these confidence measures can. This is because the former are based on reference translations, while the latter is not.

New Confidence Measures for Statistical Machine Translation

“…When it comes to natural language processing, it has been intensively studied for automatic speech recognition (Mauclair, 2006;Razik, 2007;Guo et al, 2004). We find in literature (Blatz et al, 2003;Ueffing and Ney, 2004;Ueffing and Ney, 2005;Uhrik and Ward, 1997;Akiba et al, 2004;Duchateau et al, 2002) different ways of approximating the probability of correctness or of calculating scores which are supposed to reflect this probability.…”

Word-Level Confidence Estimation for Machine Translation

“…The author found that using a small amount of manually labeled training data yields better performance than using large quantities of automatically labeled data. Akiba et al (2004) reported the application of confidence measures to the selection of output on N-best lists produced by different MT systems. Word-level confidence measures, namely the rank-weighted sum as described in Section 4.1 (and first introduced in Ueffing, Macherey and Ney [2003]), are used to discard low-quality system output before selecting a translation from the various MT systems.…”

“…In this article, we will develop a sound theoretical framework for calculating and evaluating word confidence measures. Possible applications of confidence measures include: r marking words with low confidence as potential errors for post-editing r improving translation prediction accuracy in TransType-style interactive machine translation (Gandrabur and Foster 2003;Ueffing and Ney 2005a) r combining output from different machine translation systems: Hypotheses with low confidence can be discarded before selecting one of the system translations (Akiba et al 2004), or the word confidence scores can be used in the generation of new hypotheses from the output of different systems (Jayaraman and Lavie 2005), or the sentence confidence value can be employed for reranking (Blatz et al 2003).…”

Word-level confidence estimation for machine translation using phrase-based translation models

“…• post-editing, where words with low confidence could be marked as potential errors, • improving translation prediction accuracy in trans-type-style interactive machine translation Ueffing and Ney, 2005), • combining output from different machine translation systems: hypotheses with low confidence can be discarded before selecting one of the system translations (Akiba et al, 2004), or the word confidence scores can be used for generating new hypotheses from the output of different systems (Jayaraman and Lavie, 2005), or the sentence confidence value can be employed for re-ranking (Blatz et al, 2003).…”