Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings

Abstract

Unsupervised cross-lingual word embedding (CLWE) methods learn a linear transformation matrix that maps two monolingual embedding spaces that are separately trained with monolingual corpora. This method relies on the assumption that the two embedding spaces are structurally similar, which does not necessarily hold true in general. In this paper, we argue that using a pseudo-parallel corpus generated by an unsupervised machine translation model facilitates the structural similarity of the two embedding spaces and improves the quality of CLWEs in the unsupervised mapping method. We show that our approach outperforms other alternative approaches given the same amount of data, and, through detailed analysis, we show that data augmentation with the pseudo data from unsupervised machine translation is especially effective for mappingbased CLWEs because (1) the pseudo data makes the source and target corpora (partially) parallel; (2) the pseudo data contains information on the original language that helps to learn similar embedding spaces between the source and target languages.

Data Augmentation Approaches in Natural Language Processing: A Survey

“…Paraphrasing Thesauruses Zhang et al [5], Wei et al [6], Coulombe et al [7] Semantic Embeddings Wang et al [8] MLMs Jiao et al [9] Rules Coulombe et al [7], Regina et al [10], Louvan et al [11] Machine Translation Back-translation Xie et al [12], Zhang et al [13] Unidirectional Translation Nishikawa et al [14], Bornea et al [15] Model Generation Hou et al [16], Li et al [17], Liu et al [18] Noising Swapping Wei et al [6], Luque et al [19], Yan et al [20] Deletion Wei et al [6], Peng et al [21], Yu et al [22] Insertion Wei et al [6], Peng et al [21], Yan et al [20] Substitution Coulombe et al [7], Xie et al [23], Louvan et al [11] Mixup Guo et al [24], Cheng et al [25] Sampling Rules Min et al [26], Liu et al [27] Seq2Seq Models Kang et al [28], Zhang et al [13], Raille et al [29] Language Models…”

“…In the task of unsupervised cross-lingual word embeddings (CLWEs), Nishikawa et al [14] build pseudo-parallel corpus with an unsupervised machine translation model. The authors first train unsupervised machine translation (UMT) models using the source/target training corpora and then translate the corpora using the UMT models.…”

“…That is, for a piece of text input, output the category to which the text belongs, where the category is a pre-defined closed set. 14 • Text generation, as the name implies, is to generate the corresponding text given the input data. The most classic example is machine translation.…”