Differences between SMT and NMT Output - a Translators’ Point of View

Abstract

In this study, we compare the output quality of two MT systems, a statistical (SMT) and a neural (NMT) engine, customised for Swiss Post's Language Service using the same training data. We focus on the point of view of professional translators and investigate how they perceive the differences between the MT output and a human reference (namely deletions, substitutions, insertions and word order). Our findings show that translators more frequently consider these differences to be errors in SMT than NMT, and that deletions are the most serious errors in both architectures. We also observe there to be less agreement on differences to be corrected in NMT than SMT, suggesting that errors are easier to identify in SMT. These findings confirm the ability of NMT to produce correct paraphrases, which could also explain why BLEU is often considered to be an inadequate metric to evaluate the performance of NMT systems.

Speech and Translation Technologies

“…This useful verification source has not previously been exploited. Experiments supplying confirmatory backtranslations via neural networks appear promising as well (Mutal et al, 2019; DeepL commercial translation system 2 ). For comparison, other feedback sources which have been used to date include semantically controlled backtranslation (Seligman and Dillinger, 2016) and paraphrases generated via interlingua-based semantic representations (Gao et al, 2006).…”

HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation

“…Post-editing and translation error categorization using professional translators have been carried out by researchers in assessing neural MT (NMT) outputs in recent studies (Bentivogli et al, 2016;Castilho et al, 2017;Esperanc ¸a-Rodier and Rossi, 2019;Mutal et al, 2019). For instance, Bentivogli et al (2016) argued that on a case study of translation quality on English-to-German language pair using the data from IWSLP2015, LSTM based NMT with attention model produces translation output that improves word order in placement of verbs with a large winning margin in comparison to traditional phrase-based statistical MT (PBSMT) model.…”

Enhancing Speech Translation in Medical Emergencies with Pictographs

“…These enable doctors to speak freely, with the system linking the recognition result to the closest source-language match that is a clear and explicit variant of the original sentence. This intermediate result can be presented to the doctor for confirmation, and can also be used as the input for translation into the system's target languages (Mutal et al, 2019;Bouillon et al, 2021).…”