Results of the WMT15 Metrics Shared Task

Stanojević, Miloš; Kamran, Amir; Koehn, Philipp; Bojar, Ondřej

doi:10.18653/v1/w15-3031

Cited by 65 publications

(63 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A main venue for evaluation of MT metrics is the annual Workshop for Statistical Machine Translation (WMT) where large-scale human evaluation takes place, primarily for the purpose of ranking systems competing in the translation shared task, but additionally to use the resulting system rankings for evaluation of automatic metrics. Since 2014, WMT has used the Pearson correlation as the official measure for evaluation of metrics (Macháček and Bojar, 2014;Stanojević et al, 2015). Comparison of the performance of any two metrics involves the comparison of two Pearson correlation point estimates computed over a sample of MT systems, therefore.…”

Section: Introductionmentioning

confidence: 99%

Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics

Graham

2016

Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Automatic Machine Translation metrics, such as BLEU, are widely used in empirical evaluation as a substitute for human assessment. Subsequently, the performance of a given metric is measured by its strength of correlation with human judgment. When a newly proposed metric achieves a stronger correlation over that of a baseline, it is important to take into account the uncertainty inherent in correlation point estimates prior to concluding improvements in metric performance. Confidence intervals for correlations with human judgment are rarely reported in metric evaluations, however, and when they have been reported, the most suitable methods have unfortunately not been applied. For example, incorrect assumptions about correlation sampling distributions made in past evaluations risk over-estimation of significant differences in metric performance. In this paper, we provide analysis of each of the issues that may lead to inaccuracies before providing detail of a method that overcomes previous challenges. Additionally, we propose a new method of translation sampling that in contrast achieves genuine high conclusivity in evaluation of the relative performance of metrics.

show abstract

Section: Introductionmentioning

confidence: 99%

Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics

Graham

2016

Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…We see in Tables 1 and 2 that our models that use images directly to initialise either the encoder or the decoder are the only ones to consistently outperform the PBSMT baseline according to the chrF3 metric, a character-based metric that in- cludes both precision and recall, and has a recall bias. That is also a noteworthy finding, since chrF3 is the only character-level metric we use, and it has shown a high correlation with human judgements (Stanojević et al, 2015).…”

Section: Resultsmentioning

confidence: 55%

Incorporating Global Visual Features into Attention-based Neural Machine Translation.

Calixto¹,

Li²

2017

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

117

View full text Add to dashboard Cite

We introduce multi-modal, attentionbased Neural Machine Translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. Global image features are extracted using a pre-trained convolutional neural network and are incorporated (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. In our experiments, we evaluate translations into English and German, how different strategies to incorporate global image features compare and which ones perform best. We also study the impact that adding synthetic multi-modal, multilingual data brings and find that the additional data have a positive impact on multi-modal models. We report new state-of-the-art results and our best models also significantly improve on a comparable Phrase-Based Statistical MT (PBSMT) model trained on the Multi30k data set according to all metrics evaluated. To the best of our knowledge, it is the first time a purely neural model significantly improves over a PBSMT model on all metrics evaluated on this data set.

show abstract

“…The evaluation metrics are correlated with human rankings by means of Spearman's rank correlation coefficient for the WMT13 task (Macháček and Bojar, 2013) and Pearson product-moment correlation coefficient for the WMT14 task (Macháček and Bojar, 2014) and WMT15 task (Stanojević et al, 2015) on the system level. Through the experiments we aim to investigate the following points:…”

Section: Methodsmentioning

confidence: 99%

“…The best results in each direction are in bold. We calculated the CharacTER and CHRF3 scores and cited the other scores from the WMT metric papers (Macháček and Bojar, 2013;Macháček and Bojar, 2014;Stanojević et al, 2015). * English→German scores are not included in the averages of the WMT14 metrick task.…”

Section: Comparison With Other Metricsmentioning

confidence: 99%

CharacTer: Translation Edit Rate on Character Level

Wang¹,

Peter²,

Rosendahl³

et al. 2016

Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers

View full text Add to dashboard Cite

Recently, the capability of character-level evaluation measures for machine translation output has been confirmed by several metrics. This work proposes translation edit rate on character level (CharacTER), which calculates the character level edit distance while performing the shift edit on word level. The novel metric shows high system-level correlation with human rankings, especially for morphologically rich languages. It outperforms the strong CHRF by up to 7% correlation on different metric tasks. In addition, we apply the hypothesis sentence length for normalizing the edit distance in CharacTER, which also provides significant improvements compared to using the reference sentence length.

show abstract

Results of the WMT15 Metrics Shared Task

Cited by 65 publications

References 18 publications

Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics

Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics

Incorporating Global Visual Features into Attention-based Neural Machine Translation.

CharacTer: Translation Edit Rate on Character Level

Contact Info

Product

Resources

About