Findings of the 2017 Conference on Machine Translation (WMT17)

Bojar, Ondřej; Chatterjee, Rajen; Federmann, Christian; Graham, Yvette; Haddow, Barry; Huang, Shujian; Huck, Matthias; Koehn, Philipp; Liu, Qun; Logacheva, Varvara; Monz, Christof; Negri, Matteo; Post, Matt; Rubino, Raphaël; Specia, Lucia; Turchi, Marco

doi:10.18653/v1/w17-4717

Cited by 292 publications

(268 citation statements)

References 61 publications

Supporting

Mentioning

263

Contrasting

Unclassified

Order By: Relevance

“…Next, we discuss the compilation of German-English and English-German corpora. We select these pairs, as they are among the most studied in MT, and comparatively high results are obtained for them (Bojar et al, 2017). Hence, they are more likely to benefit from a fine-grained analysis.…”

Section: A Test Case On Extracting Setsmentioning

confidence: 99%

Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

Choshen¹,

Abend²

2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

We show that the state-of-the-art Transformer MT model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, longdistance dependencies remain a challenge for the model. Since most dependencies are shortdistance, common evaluation metrics will be little influenced by how well systems perform on them. We therefore propose an automatic approach for extracting challenge sets replete with long-distance dependencies, and argue that evaluation using this methodology provides a complementary perspective on system performance. To support our claim, we compile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow reliable automatic evaluation, which makes the proposed approach a scalable and practical solution for evaluating MT performance on the long-tail of syntactic phenomena. 1

show abstract

Section: A Test Case On Extracting Setsmentioning

confidence: 99%

Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

Choshen¹,

Abend²

2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

show abstract

“…We use English translations of the Chinese source texts in the WMT 2017 English-Chinese test set (Bojar et al, 2017) for all experiments presented in this article:…”

Section: Translationsmentioning

confidence: 99%

A Set of Recommendations for Assessing Human–Machine Parity in Language Translation

Läubli

Castilho

Neubig

et al. 2020

jair

View full text Add to dashboard Cite

The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations. We reassess Hassan et al.'s 2018 investigation into Chinese to English news translation, showing that the finding of human-machine parity was owed to weaknesses in the evaluation design-which is currently considered best practice in the field. We show that the professional human translations contained significantly fewer errors, and that perceived quality in human evaluation depends on the choice of raters, the availability of linguistic context, and the creation of reference translations. Our results call for revisiting current best practices to assess strong machine translation systems in general and human-machine parity in particular, for which we offer a set of recommendations based on our empirical findings.

show abstract

“…Mean standardized scores for translation task participating systems were computed by firstly taking the average of scores for individual translations in the test set (since some were assessed more than once), before combining all scores for translations attributed to a given MT system into its overall adequacy score. The gold standard for system-level DA evaluation is thus what is denoted "Ave z" in Findings 2017 (Bojar et al, 2017a).…”

Section: System-level Manual Quality Judgmentsmentioning

confidence: 99%

“…The metrics task itself then needs manual judgements of translation quality in order to check the extent to which the automatic metrics can approximate the judgement. For situations where the reference translation is not available, please consult the results of Quality Estimation Task (Bojar et al, 2017a).…”

Section: Introductionmentioning

confidence: 99%

“…The underlying texts and MT systems come from two other WMT tasks, namely News Translation Task (Bojar et al, 2017a, denoted as Findings 2017 in the following) and Neural MT training task (Bojar et al, 2017b), and from the EU project HimL, aiming at translation of healthrelated documents. The texts were drawn mainly from the news domain and, to a limited extent, from the medical domain and involve translations to/from Chinese (zh), Czech (cs), Finnish (fi), German (de), Latvian (lv), Russian (ru), and Turkish (tr), each paired with English, and additionally English into Romanian and Polish, making a total of 16 language pairs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Results of the WMT17 Metrics Shared Task

Bojar¹,

Graham²,

Kamran³

2017

Proceedings of the Second Conference on Machine Translation

Self Cite

View full text Add to dashboard Cite

This paper presents the results of the WMT17 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT17 news translation task and Neural MT training task. We collected scores of 14 metrics from 8 research groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric's scores correlate with WMT17 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in judging the quality of a particular sentence).This year, we build upon two types of manual judgements: direct assessment (DA) and HUME manual semantic judgements.

show abstract

Findings of the 2017 Conference on Machine Translation (WMT17)

Cited by 292 publications

References 61 publications

Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

A Set of Recommendations for Assessing Human–Machine Parity in Language Translation

Results of the WMT17 Metrics Shared Task

Contact Info

Product

Resources

About