Word Confidence Estimation for SMT N-best List Re-ranking

Abstract

This paper proposes to use Word Confidence Estimation (WCE) information to improve MT outputs via N-best list reranking. From the confidence label assigned for each word in the MT hypothesis, we add six scores to the baseline loglinear model in order to re-rank the N-best list. Firstly, the correlation between the WCE-based sentence-level scores and the conventional evaluation scores (BLEU, TER, TERp-A) is investigated. Then, the N-best list re-ranking is evaluated over different WCE system performance levels: from our real and efficient WCE system (ranked 1st during last WMT 2013 Quality Estimation Task) to an oracle WCE (which simulates an interactive scenario where a user simply validates words of a MT hypothesis and the new output will be automatically re-generated). The results suggest that our real WCE system slightly (but significantly) improves the baseline while the oracle one extremely boosts it; and better WCE leads to better MT quality.

Confidence-based Rewriting of Machine Translation Output

“…We have shown that the quality of phrase-level confidence estimates has a direct impact of the amplitude of the improvements that can be obtained, as well as the initial quality of the rewritten hypotheses. We have used a very simple definition for confidence estimates under the form of phrase posteriors estimated from n-best lists from an initial decoder, which obtained good empirical performance, in spite of not requiring large human-annotated datasets as in other approaches (Bach et al, 2011;Luong et al, 2014b).…”

“…The difference between our approach and the reranking baseline lies in the manner in which we expand our training data, as well as in our use of high-confidence rewritings to obtain new translation hypotheses. Importantly, this work will only exploit simple confidence estimates corresponding to phrase-based posteriors, which do not require that large sets of human-annotated data be available as in other works (Bach et al, 2011;Luong et al, 2014b). The remainder of this paper is organized as follows.…”

“…Gimpel et al (2013) outperformed n-best reranking by generating, with an expensive but simple method, diverse hypotheses used as training data. Recently, Luong et al (2014b) reranked n-best lists using confidence scores at the hypothesis level computed from word-level confidence measures learnt from roughly 10,000 SMT system outputs annotated by humans. et al (2007) described a greedy search decoder, first introduced in (Germann et al, 2001), able to improve translations produced by a dynamic programming decoder using the same scoring function and translation table.…”

“…Many works have tackled the issue of word to n-gram confidence estimation in SMT output (Zens and Ney, 2006;Ueffing and Ney, 2007;Bach et al, 2011;de Gispert et al, 2013), and some attempts have been made to exploit confidence estimates for lattice rescoring (Blackwood et al, 2010) or n-best reranking (Bach et al, 2011;Luong et al, 2014b).…”

Automatic quality estimation for speech translation using joint ASR and MT features

“…For speech-to-text applications, CE may tell us if output translations are worth being corrected or if they require retranslation from scratch. Moreover, an accurate CE can also help to improve SLT itself through a second-pass N-best list re-ranking or search graph re-decoding, as it has already been done for text translation in [2] and [19], or for speech translation in [4]. Consequently, building a method which is capable of pointing out the correct parts as well as detecting the errors in a speech translated output is crucial to tackle above issues.…”