Do not neglect related languages: The case of low-resource Occitan cross-lingual word embeddings

Abstract

Cross-lingual word embeddings (CLWEs) have proven indispensable for various natural language processing tasks, e.g., bilingual lexicon induction (BLI). However, the lack of data often impairs the quality of representations. Various approaches requiring only weak crosslingual supervision were proposed, but current methods still fail to learn good CLWEs for languages with only a small monolingual corpus. We therefore claim that it is necessary to explore further datasets to improve CLWEs in low-resource setups. In this paper we propose to incorporate data of related high-resource languages. In contrast to previous approaches which leverage independently pre-trained embeddings of languages, we (i) train CLWEs for the low-resource and a related language jointly and (ii) map them to the target language to build the final multilingual space. In our experiments we focus on Occitan, a low-resource Romance language which is often neglected due to lack of resources. We leverage data from French, Spanish and Catalan for training and evaluate on the Occitan-English BLI task. By incorporating supporting languages our method outperforms previous approaches by a large margin. Furthermore, our analysis shows that the degree of relatedness between an incorporated language and the low-resource language is critically important.

Isomorphic Cross-lingual Embeddings for Low-Resource Languages

“…Across all language pairs, we see substantial gains from our method as compared to mapping, joint, and other hybrid baselines. (Woller et al, 2021) outperform our approach on two language pairs (Ne-En, Gu-En) which we suspect is due to Joint-Align's performance in comparison to regular Joint training. However, these improvements are not consistent.…”

“…Although our proposed framework does not make any significant changes to the mapping and joint components, the combination of the two cross-lingual approaches leads to better embeddings both in terms of quality, shown by the performance in BLI, as well as structure, shown by the eigenvalue similarity scores. In addition to this, MGPA as well as (Woller et al, 2021)'s method attains good eigenvalue similarity scores suggesting that the incorporation of a related language is indeed helpful…”

“…Furthermore, for low-resource setups (Kementchedjhieva et al, 2018) propose Multi-support Generalized Procrustes Analysis (MGPA) which learns a threeway alignment between English, a low-resource language, and a supporting related language. In addition to (Wang et al, 2020), (Woller et al, 2021) show the benefit of using related languages in the context of CLWEs. In their work, they use a two step approach where they first use joint-align to learn a CLWE between the low-resource and related language and as a next step they map it to the target language to build the final multilingual embedding space.…”

“…We use the Low-Resource to English language direction however MGPA can only be trained with the related and low-resource language at the target. As per (Woller et al, 2021), we evaluate the resultant Low-Resource-English embeddings afterwards. Across all language pairs, we see substantial gains from our method as compared to mapping, joint, and other hybrid baselines.…”

“…In eigenvalue similarity, mapping methods perform much worse than joint training (Table 5). This finding is in line with the literature (Ormazabal et al, 2019), and is explained by the high linguistic divergence between English and source languages, ne → en fi → en ro → en gu → en hu → en avg VecMap (Artetxe et al, 2018b) 52.3 61.9 61.6 45.4 53.2 54.8 Joint (Luong et al, 2015) 21.3 30.5 31.4 33.4 25.5 28.4 Joint Align (Wang et al, 2020) 24.5 31.3 28.2 35.4 26.5 25.2 MGPA (Kementchedjhieva et al, 2018) 41.6 55.7 57.3 39.6 45.7 47.9 Joint Align + MUSE (Woller et al, 2021) resulting in embeddings that are far less isomorphic. Our hybrid approach performs even better than joint methods and achieved the best eigenvalue similarity score across all language pairs, showing that we do indeed obtain isometric embeddings while still not requiring the higher level of supervision in joint learning approaches.…”