Is Neural Machine Translation the New State of the Art?

Castilho, Sheila; Moorkens, Joss; Gaspari, Federico; Calixto, Iacer; Tinsley, John; Way, Andy

doi:10.1515/pralin-2017-0013

Cited by 152 publications

(88 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As the issues are complementary between the two MT paradigms analysed, results suggest that hybridisation between them could be a promising way forward. -Castilho et al (2017) evaluated the performance of NMT versus PBMT for three different translation domains: e-commerce product listings, patents and massive open online courses. They performed error analysis with an error taxonomy consisting of 7 categories for patent translation from Chinese to English.…”

Section: Introductionmentioning

confidence: 99%

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Klubička¹,

Toral

Sánchez-Cartagena³

2018

Machine Translation

View full text Add to dashboard Cite

This paper presents a quantitative fine-grained manual evaluation approach to comparing the performance of different machine translation (MT) systems. We build upon the well-established Multidimensional Quality Metrics (MQM) error taxonomy and implement a novel method that assesses whether the differences in performance for MQM error types between different MT systems are statistically significant. We conduct a case study for English-toCroatian, a language direction that involves translating into a morphologically rich language, for which we compare three MT systems belonging to different paradigms: pure phrase-based, factored phrase-based and neural. First, we design an MQM-compliant error taxonomy tailored to the relevant linguistic phenomena of Slavic languages, which made the annotation process feasible and accurate. Errors in MT outputs were then annotated by two annotators following this taxonomy. Subsequently, we carried out a statistical analysis which showed that the best-performing system (neural) reduces the errors produced by the worst system (pure phrase-based) by more than half (54%). Moreover, we conducted an additional analysis of agreement errors in which we distinguished between short (phrase-level) and long distance (sentence-level) errors. We discovered that phrase-based MT approaches are of limited use for long distance agreement phenomena, for which neural MT was found to be especially effective.

show abstract

Section: Introductionmentioning

confidence: 99%

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Klubička¹,

Toral

Sánchez-Cartagena³

2018

Machine Translation

View full text Add to dashboard Cite

show abstract

“…This method underlies most of today's MT technology, where computer programs attempt to emulate human translators' decisions by learning patterns from large quantities of bilingual texts. Even some of the most recent of these technologies have limitations (see Castilho et al 2017). However, the popular press has been quick to declare that the "language barrier is about to fall" (Ross 2016).…”

Section: Work Automation Trends and Translationmentioning

confidence: 99%

Automation anxiety and translators

Vieira

2018

Translation Studies

View full text Add to dashboard Cite

Translation is currently described as a profession under pressure from automation, falling prices and globalized competition. Translators' stance on machine translation (MT) is famously negative, but the economic dimension of this positioning is scarcely researched and often unclear. This article provides an analysis of translators' blog and forum postings contextualized within general trends in employment, the economy and work automation. The analysis concentrates on MT and pay. Two key findings are reported. First, MT was found to be a secondary issue in translators' comments on pay; most grievances were based on business practices themselves. Second, most criticisms of MT were rooted not in fears of being outperformed by MT systems, but rather in the technology's limitations and market consequences. This article calls for a broadening of translators' role across areas of specialization and argues that, in the debate on translation's future, MT cannot be decoupled from its economic effects.

show abstract

“…The advantages of NMT and its challenges have been investigated from different angles in recent work (Koehn and Knowles, 2017;Toral and Sánchez-Cartagena, 2017;Farajian et al, 2017;Macketanz et al, 2017;Castilho et al, 2017a;Klubička et al, 2017;Bentivogli et al, 2018;Isabelle et al, 2017;Popović, 2017;Forcada, 2017;Castilho et al, 2017b;Junczys-Dowmunt et al, 2016;Klubička et al, 2018;Shterionov et al, 2017;Burchardt et al, 2017, inter alia), however, studies on interactive NMT, especially user studies involving human post-edits of NMT outputs, have so far not been presented.…”

Section: Related Workmentioning

confidence: 99%

A user-study on online adaptation of neural machine translation to human post-edits

Karimova

Simianer

Riezler

2018

Machine Translation

View full text Add to dashboard Cite

The advantages of neural machine translation (NMT) have been extensively validated for offline translation of several language pairs for different domains of spoken and written language. However, research on interactive learning of NMT by adaptation to human post-edits has so far been confined to simulation experiments. We present the first user study on online adaptation of NMT to user post-edits in the domain of patent translation. Our study involves 29 human subjects (translation students) whose post-editing effort and translation quality were measured on about 4,500 interactions of a human post-editor and a machine translation system integrating an online adaptive learning algorithm. Our experimental results show a significant reduction of human post-editing effort due to online adaptation in NMT according to several evaluation metrics, including hTER, hBLEU, and KSMR. Furthermore, we found significant improvements in BLEU/TER between NMT outputs and professional translations in granted patents, providing further evidence for the advantages of online adaptive NMT in an interactive setup.

show abstract

Is Neural Machine Translation the New State of the Art?

Cited by 152 publications

References 13 publications

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Automation anxiety and translators

A user-study on online adaptation of neural machine translation to human post-edits

Contact Info

Product

Resources

About