Predicting success in machine translation

Birch, Alexandra; Osborne, Miles; Koehn, Philipp

doi:10.3115/1613715.1613809

Cited by 46 publications

(55 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We determine the probability of the orientation type for the constituent pairs based on Equation (1). Based on the orientation class for a pair, we consider its score for calculating monotone or swap feature functions and compute F monotone and F swap as shown in Equation (4), where a is the word alignment between S and H, and Aligned(H, Pairs(S), a) shows the aligned covered pairs based on the word alignment a.…”

Section: Integration Into Hpb-smtmentioning

confidence: 99%

“…Accordingly, in order to translate a sentence from the source language into the target language, SMT has to handle two problems: (i) finding the appropriate translation of the words in the source sentence ("lexical choice"), and (ii) predicting their correct order in the target sentence ("reordering"). Reordering is one of the most important factors affecting the quality of the final translation [1]. A large amount of research has been conducted to address the reordering problem, much of which follows the discriminative reordering model (DRM), i.e., they consider word reordering as an structured prediction problem and apply a discriminatively trained model to predict the appropriate word order.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Investigating the Relationship between Classification Quality and SMT Performance in Discriminative Reordering Models

Kazemi

Toral

Way

et al. 2017

Entropy

View full text Add to dashboard Cite

Abstract:Reordering is one of the most important factors affecting the quality of the output in statistical machine translation (SMT). A considerable number of approaches that proposed addressing the reordering problem are discriminative reordering models (DRM). The core component of the DRMs is a classifier which tries to predict the correct word order of the sentence. Unfortunately, the relationship between classification quality and ultimate SMT performance has not been investigated to date. Understanding this relationship will allow researchers to select the classifier that results in the best possible MT quality. It might be assumed that there is a monotonic relationship between classification quality and SMT performance, i.e., any improvement in classification performance will be monotonically reflected in overall SMT quality. In this paper, we experimentally show that this assumption does not always hold, i.e., an improvement in classification performance might actually degrade the quality of an SMT system, from the point of view of MT automatic evaluation metrics. However, we show that if the improvement in the classification performance is high enough, we can expect the SMT quality to improve as well. In addition to this, we show that there is a negative relationship between classification accuracy and SMT performance in imbalanced parallel corpora. For these types of corpora, we provide evidence that, for the evaluation of the classifier, macro-averaged metrics such as macro-averaged F-measure are better suited than accuracy, the metric commonly used to date.

show abstract

Section: Integration Into Hpb-smtmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Investigating the Relationship between Classification Quality and SMT Performance in Discriminative Reordering Models

Kazemi

Toral

Way

et al. 2017

Entropy

View full text Add to dashboard Cite

show abstract

“…We randomly select 40 sentences which have a large amount of reordering (RQuantity (Birch et al 2008) > 1.3) and where the sentence length is between 10 and 40 words.…”

Section: Design and Materialsmentioning

confidence: 99%

“…The word order differences are difficult to model and the amount of reordering has been shown to be a very important predictive factor in translation performance (Birch et al 2008). We argue that research in reordering is hampered by the fact that commonly used automatic metrics do not explicate the reordering performance of machine translation systems.…”

Section: Introductionmentioning

confidence: 98%

Metrics for MT evaluation: evaluating reordering

Birch

Osborne

Blunsom

2010

Machine Translation

Self Cite

View full text Add to dashboard Cite

Translating between dissimilar languages requires an account of the use of divergent word orders when expressing the same semantic content. Reordering poses a serious problem for statistical machine translation systems and has generated a considerable body of research aimed at meeting its challenges. Direct evaluation of reordering requires automatic metrics that explicitly measure the quality of word order choices in translations. Current metrics, such as BLEU, only evaluate reordering indirectly. We analyse the ability of current metrics to capture reordering performance. We then introduce permutation distance metrics as a direct method for measuring word order similarity between translations and reference sentences. By correlating all metrics with a novel method for eliciting human judgements of reordering quality, we show that current metrics are largely influenced by lexical choice, and that they are not able to distinguish between different reordering scenarios. Also, we show that permutation distance metrics correlate very well with human judgements, and are impervious to lexical differences.

show abstract

“…Automatically identifying such phrases has the potential of improving MT as shown by an oracle study (Mohit and Hwa, 2007). More recent work (Birch et al, 2008) has shown that properties of reordering, source and target language complexity and relatedness can be used to predict translation quality. In information retrieval, the problem of predicting system performance has generated considerable interest and has led to notably good results (Cronen-Townsend et al, 2002;Yom-Tov et al, 2005;Carmel et al, 2006).…”

Section: Introductionmentioning

confidence: 99%

Performance confidence estimation for automatic summarization

Louis¹,

Nenkova²

2009

Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on - EACL '09

View full text Add to dashboard Cite

We address the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either single-or multi-document inputs. Our labelled corpus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We examine different formulations of the classification task which considerably influence performance. The best results are 84% prediction accuracy for single-and 74% for multi-document summarization. AbstractWe address the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either single-or multi-document inputs. Our labelled corpus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We examine different formulations of the classification task which considerably influence performance. The best results are 84% prediction accuracy for single-and 74% for multi-document summarization.

show abstract

Predicting success in machine translation

Cited by 46 publications

References 10 publications

Investigating the Relationship between Classification Quality and SMT Performance in Discriminative Reordering Models

Investigating the Relationship between Classification Quality and SMT Performance in Discriminative Reordering Models

Metrics for MT evaluation: evaluating reordering

Performance confidence estimation for automatic summarization

Contact Info

Product

Resources

About