The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Translation

Abstract

Supervised Neural Machine Translation (NMT) systems currently achieve impressive translation quality for many language pairs. One of the key features of a correct translation is the ability to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. Existing evaluation benchmarks on WSD capabilities of translation systems rely heavily on manual work and cover only few language pairs and few word types. We present MU-COW, a multilingual contrastive test suite that covers 16 language pairs with more than 200 000 contrastive sentence pairs, automatically built from word-aligned parallel corpora and the wide-coverage multilingual sense inventory of BabelNet. We evaluate the quality of the ambiguity lexicons and of the resulting test suite on all submissions from 9 language pairs presented in the WMT19 news shared translation task, plus on other 5 language pairs using pretrained NMT models. The MUCOW test suite is available at http://github. com/Helsinki-NLP/MuCoW.

Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English

“…Table 4 gives the best accuracy on in-domain and out-of-domain test sets. The accuracy of both models has a substantial drop on the out-of-domain test set which is consistent with the finding from Raganato et al (2019). The drop of char-d7 is even bigger than that of bpe-d7 which indicates that CHAR models are not more robust to domain-mismatch when learning word senses compared to BPE-based models.…”

“…For the WSD probing task, we use the FI-EN part of the MuCoW (Raganato et al, 2019) test set, which is a multilingual test suite for WSD in the WMT19 shared task. It has 2,117 annotated sentences.…”

Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

“…In this respect, a model with predefined fixed patterns may struggle to encode global semantic features. To this end, we evaluate our models on two German-English WSD test suites, ContraWSD (Rios Gonzales et al, 2017) and MuCoW (Raganato et al, 2019). 9 Table 6 shows the performance of our models on the WSD benchmarks.…”

Detecting Word Sense Disambiguation Biases in Machine Translation for Model-Agnostic Adversarial Attacks

“…We next compile a parallel lexicon of homograph translations, prioritizing a high coverage of all possible senses. Similar to (Raganato et al, 2019), we obtain sense-specific translations from crosslingual BabelNet (Navigli and Ponzetto, 2010) synsets. Since BabelNet entries vary in their granularity, we iteratively merge related synsets as long as they have at least three German translations in common or share at least one definition.…”