Comparing Language Related Issues for NMT and PBMT between German and English

Abstract

This work presents an extensive comparison of language related problems for neural machine translation and phrase-based machine translation between German and English. The explored issues are related both to the language characteristics as well as to the machine translation process and, although related, are going beyond typical translation error classes. It is shown that the main advantage of the NMT system consists of better handling of verbs, English noun collocations, German compound words, phrase structure as well as articles. In addition, it is shown that the main obstacles for the NMT system are prepositions, translation of English (source) ambiguous words and generating English (target) continuous tenses. Although in total there are less issues for the NMT system than for the PBMT system, many of them are complementary -only about one third of the sentences deals with the same issues, and for about 40% of the sentences the issues are completely different. This means that combination/hybridisation of the NMT and PBMT approaches is a promising direction for improving both types of systems.

Proceedings of the Third Workshop on Discourse in Machine Translation

“…While in 2015 we used macro-averaged F 1 as an official evaluation measure, this year we followed the setup of 2016, where we switched to macroaveraged recall, which was also recently adopted by some other competitions, e.g., by SemEval-2016/2017 Task 4 (Nakov et al, 2016 Rosenthal et al, 2017). Moreover, as in 2015 and 2016, we also report accuracy as a secondary evaluation measure (but we abandon F 1 altogether).…”

“…As part of her analysis Popović (2017) attempts to classify the type of errors made by each system. A most constructive development, Burchardt et al (2017) introduces a test suite which while it is common and invaluable in software engineering, is not widespread for this domain.…”

“…With the success of NMT and the changed architecture it brings, embrace this opportunity to advance to a deeper level of translation. As illustrated by recent comparative research into output from Phrase Based Machine Translation (PBMT) and NMT systems (Popović, 2017;Burchardt et al, 2017), the latter is capable of producing output which is far more linguistically informed.It would seem a good time to revisit the basics of translation theory, with a view to taking MT to a deeper level. …”

Editors’ foreword to the special issue on human factors in neural machine translation

“…Error analysis of NMT systems has also been on the radar of the MT field. Several papers have carried out automatic (Bentivogli et al 2016;Toral and Sánchez-Cartagena 2017) or human error annotation (Burchardt et al 2017;Klubička et al 2017;Popović 2017;Castilho et al 2018) in order to compare phrase-based and neural approaches for different language pairs and domains. In this issue, Calixto and Liu present an extensive error analysis of several MT systems, including two text-only systems that fall into the PBSMT and NMT paradigms, and a set of multimodal NMT models which use not only text but also visual information extracted from images.…”

On Integrating Discourse in Machine Translation

“…With the success of NMT and the changed architecture it brings, embrace this opportunity to advance to a deeper level of translation. As illustrated by recent comparative research into output from Phrase Based Machine Translation (PBMT) and NMT systems (Popović, 2017;Burchardt et al, 2017), the latter is capable of producing output which is far more linguistically informed.…”