Translation Quality and Error Recognition in Professional Neural Machine Translation Post-Editing

Abstract

This study aims to analyse how translation experts from the German department of the European Commission's Directorate-General for Translation (DGT) identify and correct different error categories in neural machine translated texts (NMT) and their post-edited versions (NMTPE). The term translation expert encompasses translator, post-editor as well as revisor. Even though we focus on neural machine-translated segments, translator and post-editor are used synonymously because of the combined workflow using CAT-Tools as well as machine translation. Only the distinction between post-editor, which refers to a DGT translation expert correcting the neural machine translation output, and revisor, which refers to a DGT translation expert correcting the post-edited version of the neural machine translation output, is important and made clear whenever relevant. Using an automatic error annotation tool and the more fine-grained manual error annotation framework to identify characteristic error categories in the DGT texts, a corpus analysis revealed that quality assurance measures by post-editors and revisors of the DGT are most often necessary for lexical errors. More specifically, the corpus analysis showed that, if post-editors correct mistranslations, terminology or stylistic errors in an NMT sentence, revisors are likely to correct the same error type in the same post-edited sentence, suggesting that the DGT experts were being primed by the NMT output. Subsequently, we designed a controlled eye-tracking and key-logging experiment to compare participants' eye movements for test sentences containing the three identified error categories (mistranslations, terminology or stylistic errors) and for control sentences without errors. We examined the three error types' effect on early (first fixation durations, first pass durations) and late eye movement measures (e.g., total reading time and regression path durations). Linear mixed-effects regression models predict what kind of behaviour of the DGT experts is associated with the correction of different error types during the post-editing process.Informatics 2019, 6, 41 2 of 29 the development of their own machine translation systems. Statistical machine translation (internally referred to as MT@EC) has been available at the DGT since 2013 and is now coming to an end. The new neural system, eTranslation, was launched in November 2017 and is able to "translate between any pair of the 24 official EU languages, as well as Icelandic and Norwegian (Bokmål): it can handle formatted documents and plain text; it translates multiple documents into multiple languages in "one go"; it accepts diverse input formats including XML and PDF; it retains formatting; and it provides specific output formats for computer-aided translation, i.e., TMX11 and XLIFF."It is trained with the aligned source and target segments that have been produced over the years-amounting to approximately 1.2 billion training segments ([1], p. 3).However, the usage of machine translation requires a different form of revisi...

Neural Machine Translation: Fine-Grained Evaluation of Google Translate Output for English-to-Arabic Translation

“…However, based on the results of previous research (e.g., Costa et al, 2015;Kafipour & Jahanshahi, 2015) regarding the challenges encountered by MT, the present study aims to investigate the quality of Google MT for the Arabic-English language pair after it was updated in 2017. Evaluating the quality of Google neural machine translation (GNMT) is a relatively new research field that has not yet been explored extensively (Vardaro et al, 2019).…”

ترجمة جوجل ومسألة الأیدیولوجیا فی عناوین الأخبار السیاسیة

“…The second area discusses the challenges translators might face, and the skills they need to learn and acquire with the introduction and the application of MT (Pym, 2013(Pym, , 2019. The third area highlights the efficacy of MT on the quality of translation; that is to say, how MT addresses translation problems on the micro and macro level (Wilks, 2009;Peng, 2018;Vardaro et al 2019). To this latter area, this present research is trying to contribute.…”

Evaluating the Impact of Integrating Similar Translations into Neural Machine Translation

“…In order for the findings to be reliable, it should be possible to apply an error analysis task consistently by multiple assessors, yielding a high inter-annotator agreement (IAA) [68]. Fine-grained error analysis has been applied to different MT architectures and it is still a common MT quality assessment technique that allows us to identify the strengths and weaknesses of NMT systems, for example, for different domains and language pairs [67,[69][70][71]. To our knowledge, NFR output, or that of similar TM-NMT integration systems, has not been analysed yet in terms of the types of errors it contains.…”