How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions

Abstract

Cross-lingual word embeddings (CLEs) enable multilingual modeling of meaning and facilitate cross-lingual transfer of NLP models. Despite their ubiquitous usage in downstream tasks, recent increasingly popular projectionbased CLE models are almost exclusively evaluated on a single task only: bilingual lexicon induction (BLI). Even BLI evaluations vary greatly, hindering our ability to correctly interpret performance and properties of different CLE models. In this work, we make the first step towards a comprehensive evaluation of cross-lingual word embeddings. We thoroughly evaluate both supervised and unsupervised CLE models on a large number of language pairs in the BLI task and three downstream tasks, providing new insights concerning the ability of cutting-edge CLE models to support cross-lingual NLP. We empirically demonstrate that the performance of CLE models largely depends on the task at hand and that optimizing CLE models for BLI can result in deteriorated downstream performance. We indicate the most robust supervised and unsupervised CLE models and emphasize the need to reassess existing baselines, which still display competitive performance across the board. We hope that our work will catalyze further work on CLE evaluation and model analysis.

Unsupervised Joint Training of Bilingual Word Embeddings

“…For enid, we used English (100M lines) and Indonesian (77M lines) Common Crawl corpora. 5 We then mapped the word embeddings into a BWE space using VECMAP, 6 one of the best and most robust methods for unsupervised mapping (Glavas et al, 2019). The resulting BWE were used as baselines in our evaluation tasks and also to bootstrap our USMT system.…”

“…Bilingual lexicon induction (BLI) is by far the most popular evaluation task for BWE used by previous work in spite of its limits (Glavas et al, 2019). In contrast to previous work, we used much larger test sets 10 for each language pair.…”

Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectors

“…Furthermore, unlike Caliskan et al (2017), we test whether biases depend on the selection of the similarity metric. Finally, given the ubiquitous adoption of cross-lingual embeddings (Ruder et al, 2017;Glavaš et al, 2019), we investigate biases in a variety of bilingual embedding spaces.…”

Cross-lingual similar document retrieval methods

“…We use word2vec skip-gram model [16] with the following parameters: dimensionality of embeddings was 300, a window size of 10 words, the minimal corpus frequency of 10, negative sampling with 10 samples, no downsampling, 20 iterations over the corpus. Then we use vecmap [17,18] framework to learn a transformation matrix that maps representations in one language to the representations of the 1…”