Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models

Abstract

We study several methods for full or partial sharing of the decoder parameters of multilingual NMT models. We evaluate both fully supervised and zero-shot translation performance in 110 unique translation directions using only the WMT 2019 shared task parallel datasets for training. We use additional test sets and re-purpose evaluation methods recently used for unsupervised MT in order to evaluate zero-shot translation performance for language pairs where no gold-standard parallel data is available. To our knowledge, this is the largest evaluation of multi-lingual translation yet conducted in terms of the total size of the training data we use, and in terms of the diversity of zero-shot translation pairs we evaluate. We conduct an in-depth evaluation of the translation performance of different models, highlighting the trade-offs between methods of sharing decoder parameters. We find that models which have task-specific decoder parameters outperform models where decoder parameters are fully shared across all tasks.

Findings of the 2019 Conference on Machine Translation (WMT19)

“…Pre-trained embedding is trained on monolingual data for 5 iterations and used as an initialization for the RNN model. (Hokamp et al, 2019) The Aylien research team built a Multilingual NMT system which is trained on all WMT2019 language pairs in all directions, then fine-tuned for a small number of iterations on Gujarati-English data only, including some self-backtranslated data.…”

“…Air Force Research Laboratory ) APERTIUM-FIN-ENG Apertium (Pirinen, 2019) APPRENTICE-C Apprentice (Li and Specia, 2019) AYLIEN_MULTILINGUAL Aylien Ltd. (Hokamp et al, 2019) BAIDU Baidu (Sun et al, 2019) BTRANS (no associated paper)…”

A Survey of Multilingual Neural Machine Translation

“…By minimizing the diversity of representations, the decoder's task is simplified and it becomes better at language generation. The choice of a single encoder for all languages is also promoted by Hokamp et al [64], who opt for language-specific decoders. Murthy et al [101] pointed out that the sentence representations generated by the encoder are dependent on the word order of the language and are, hence, language-specific.…”

“…This shows that dedicating a few parameters to learn language tokens can help a decoder maintain a balance between language-agnostic and language-distinct features. Hokamp et al [64] showed that more often than not, using separate decoders and attention mechanisms gives better results as compared to a shared decoder and attention mechanism. This work implies that the best way to handle language divergence would be to use a shared encoder for source languages and different decoders for target languages.…”

“…Each of these methods provides gains over Johnson et al [70], and combining all methods gave the best results. Hokamp et al [64] showed that in a shared decoder setting, using a taskspecific (language pair to be translated) embedding works better than using language tokens. We expect that this is because learning task-specific embeddings needs more parameters and helps the decoder learn better features to distinguish between tasks.…”