Word-Level Confidence Estimation for Machine Translation

Abstract

This article introduces and evaluates several different word-level confidence measures for machine translation. These measures provide a method for labeling each word in an automatically generated translation as correct or incorrect. All approaches to confidence estimation presented here are based on word posterior probabilities. Different concepts of word posterior probabilities as well as different ways of calculating them will be introduced and compared. They can be divided into two categories: System-based methods that explore knowledge provided by the translation system that generated the translations, and direct methods that are independent of the translation system. The system-based techniques make use of system output, such as word graphs or N-best lists. The word posterior probability is determined by summing the probabilities of the sentences in the translation hypothesis space that contains the target word. The direct confidence measures take other knowledge sources, such as word or phrase lexica, into account. They can be applied to output from nonstatistical machine translation systems as well. Experimental assessment of the different confidence measures on various translation tasks and in several language pairs will be presented. Moreover,the application of confidence measures for rescoring of translation hypotheses will be investigated.

Confidence-based Rewriting of Machine Translation Output

“…The Word Posterior Probability (WPP) proposed by Ueffing and Ney (2007), derived from information from the n-best list produced by a decoder, proved to be useful for estimating word-level confidence. Bach et al (2011) worked on the issue of predicting sentence-level and word-level MT errors by using WPP and other features derived from the source context, the source-target alignment, and dependency structures, but relied on a significantly large manually annotated corpus of MT errors.…”

Semi-supervised model adaptation for statistical machine translation

“…Both features are determined for both translation directions, -posterior probabilities for words, phrases, n-grams, and sentence length (Zens and Ney 2006;Ueffing and Ney 2007), all calculated over the N -best list and using the sentence probabilities which the baseline system assigns to the translation hypotheses.…”

“…-Confidence estimation: The goal of confidence estimation is to estimate how reliable a translation t is, given the corresponding source sentence s. The confidence estimation which we implemented follows the approaches suggested in Blatz et al (2003) and Ueffing and Ney (2007), where the confidence score of a target sentence t is calculated as a log-linear combination of several different sentence scores. These scores are Levenshtein-based word posterior probabilities, phrase posterior probabilities, and a target language model score.…”

“…The sentence probability is approximated by the probability which the SMT system assigns to the sentence pair. More details on computing word posterior probabilities are available in Ueffing and Ney (2007).…”

Statistical speech translation system based on voice recognition optimization using multimodal sources of knowledge and characteristics vectors

“…Works like the ones presented in [22], [23], [24] are based on the use of confidence measures in statistical translation machines in order to improve the error in the translation. All these papers explain how we can use characteristics vectors from these measures, obtained from the "N-best list" and the first model of IBM, a translation output can be determined from a given input.…”