Learning Unsupervised Multilingual Word Embeddings with Incremental Multilingual Hubs

Abstract

Recent research has discovered that a shared bilingual word embedding space can be induced by projecting monolingual word embedding spaces from two languages using a selflearning paradigm without any bilingual supervision. However, it has also been shown that for distant language pairs such fully unsupervised self-learning methods are unstable and often get stuck in poor local optima due to reduced isomorphism between starting monolingual spaces. In this work, we propose a new robust framework for learning unsupervised multilingual word embeddings that mitigates the instability issues. We learn a shared multilingual embedding space for a variable number of languages by incrementally adding new languages one by one to the current multilingual space. Through the gradual language addition our method can leverage the interdependencies between the new language and all other languages in the current multilingual hub/space. We find that it is beneficial to project more distant languages later in the iterative process. Our fully unsupervised multilingual embedding spaces yield results that are on par with the state-of-the-art methods in the bilingual lexicon induction (BLI) task, and simultaneously obtain state-of-the-art scores on two downstream tasks: multilingual document classification and multilingual dependency parsing, outperforming even supervised baselines. This finding also accentuates the need to establish evaluation protocols for cross-lingual word embeddings beyond the omnipresent intrinsic BLI task in future work.

Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?

“…Therefore, the follow-up work aimed to improve the robustness of unsupervised CLWE induction by introducing more robust self-learning procedures (Artetxe et al, 2018b;Kementchedjhieva et al, 2018). Besides increased robustness, recent work claims that fully unsupervised projection-based CLWEs can even match or surpass their supervised counterparts Artetxe et al, 2018b;Alvarez-Melis and Jaakkola, 2018;Hoshen and Wolf, 2018;Heyman et al, 2019).…”

A Simple Approach to Learning Unsupervised Multilingual Embeddings

“…We solve the above two stages sequentially using known techniques. Our methodology contrasts with the existing unsupervised MWE methods (Alaux et al, 2019;Chen and Cardie, 2018;Heyman et al, 2019), which learn the unsupervised word alignments and the cross-lingual word embedding mappings jointly. Despite its apparent simplicity, we empirically observe that the proposed approach illustrates remarkable generalization ability and robustness.…”

“…They obtain the bilingual lexicons using the Gromov-Wasserstein approach (Alvarez-Melis and Jaakkola, 2018) and mapping operators between languages using the RCSLS algorithm (Joulin et al, 2018). Heyman et al (2019) propose to learn the shared multilingual space by incrementally adding languages to it, one in each iteration. Their approach is based on a reformulation of the bilingual self-learning algorithm proposed by Artetxe et al (2018b).…”

“…The subproblems are separately solved using existing techniques. In contrast, existing unsupervised multilingual approaches (Chen and Cardie, 2018;Heyman et al, 2019;Alaux et al, 2019) solve the above subproblems jointly. Though it appears like a simple baseline approach, the proposed framework provides the robustness and versatility often desired while learning an effective multilingual space for distant languages, which is a challenging setting for unsupervised methods (Søgaard et al, 2018;.…”

Hierarchical Mapping for Crosslingual Word Embedding Alignment

“…Wada et al (2019) instead use a sentence-level neural language model for directly learning multilingual word embeddings and as a result bypassing the need for mapping functions. In the paradigm of aligning pre-trained word embeddings where we focus, Heyman et al (2019) propose a technique that iteratively builds a multilingual space starting from a monolingual space and incrementally incorporating languages to it. Even if this strategy deviates from the traditional TB/MP model, it still preserves the idea of having a pivot language.…”