Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach

Abstract

In the context of neural machine translation, data augmentation (DA) techniques may be used for generating additional training samples when the available parallel data are scarce. Many DA approaches aim at expanding the support of the empirical data distribution by generating new sentence pairs that contain infrequent words, thus making it closer to the true data distribution of parallel sentences. In this paper, we propose to follow a completely different approach and present a multi-task DA approach in which we generate new sentence pairs with transformations, such as reversing the order of the target sentence, which produce unfluent target sentences. During training, these augmented sentences are used as auxiliary tasks in a multi-task framework with the aim of providing new contexts where the target prefix is not informative enough to predict the next word. This strengthens the encoder and forces the decoder to pay more attention to the source representations of the encoder. Experiments carried out on six lowresource translation tasks show consistent improvements over the baseline and over DA methods aiming at extending the support of the empirical data distribution. The systems trained with our approach rely more on the source tokens, are more robust against domain shift and suffer less hallucinations.

Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translation

“…Our framework -which extends preliminary work 1 reported in a conference paper by the same authors [13]does not require elaborate preprocessing steps, training additional systems, or data besides the available training parallel corpora. Experiments with ten low-resource translation tasks show that it systematically outperforms state-of-the art methods aimed at extending the support of the empirical data distribution.…”

“…4. This differs from previous work[13] in which transformations were applied to the original training samples during the pre-processing of the corpus, and therefore before training.…”

“…For SeqMix, we also integrated the modifications in data processing and training loss published by their authors as a fairseq task. 13 Systems trained with MaTiLDA were fine-tuned on the original training samples after being trained on the combination of original and synthetic training samples. The results reported are those obtained with the model that maximizes BLEU on the development set.…”

“…1. The additional contributions of this paper with respect to the conference paper [13] are as follows: (i) more sophisticated approach for generating the synthetic samples during training; (ii) additional experiments on both low-resource and high-resource translation tasks; (iii) more exhaustive comparison to other DA methods; (iv) improved evaluation of the contribution of the source representations generated by the encoder during inference; and (v) better grounded analysis of the tendency to hallucinate of the models evaluated.…”

Masakhane Web: A Machine Translation Platform for African Languages

“…The highest BLEU of 45.98 was obtained for word-level tokenization, showing that selecting the right pre-processing methods may improve the performance of the MT models. [25] created an NMT between English and Swahili using news data. Authors obtained BLEU of 27.42 for English-to-Swahili translation when using the data that included part-of-speech tags outperforming Google translate.…”