XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Abstract

Cross-lingual pre-training has achieved great successes using monolingual and bilingual plain text corpora. However, most pre-trained models neglect multilingual knowledge, which is language agnostic but comprises abundant cross-lingual structure alignment. In this paper, we propose XLM-K, a cross-lingual language model incorporating multilingual knowledge in pre-training. XLM-K augments existing multilingual pre-training with two knowledge tasks, namely Masked Entity Prediction Task and Object Entailment Task. We evaluate XLM-K on MLQA, NER and XNLI. Experimental results clearly demonstrate significant improvements over existing multilingual language models. The results on MLQA and NER exhibit the superiority of XLM-K in knowledge related tasks. The success in XNLI shows a better cross-lingual transferability obtained in XLM-K. What is more, we provide a detailed probing analysis to confirm the desired knowledge captured in our pre-training regimen. The code is available at https://github.com/microsoft/Unicoder/tree/master/pretraining/xlmk.

EASE: Entity-Aware Contrastive Learning of Sentence Embedding

“…Our experiments have demonstrated that entity supervision in EASE improves the quality of sentence embeddings both in the monolingual setting and, in particular, the multilingual setting. As recent studies have shown, entity annotations can be used as anchors to learn quality cross-lingual representations (Calixto et al, 2021;Nishikawa et al, 2021;Jian et al, 2022;Ri et al, 2022), and our work is another demonstration of their utility, particularly in sentence embeddings. One promising future direction is exploring how to better exploit the cross-lingual nature of entities.…”

“…thus offer a useful cross-lingual alignment supervision (Calixto et al, 2021;Nishikawa et al, 2021;Jian et al, 2022;Ri et al, 2022). The extensive multilingual support of Wikipedia alleviates the need for a parallel resource to train well-aligned multilingual sentence embeddings, especially for low-resource languages.…”

Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval

“…Unicoder (Huang et al, 2019) presents several pre-training tasks upon parallel corpora and ERNIE-M (Ouyang et al, 2021) learns semantic alignment by leveraging back translation. XLM-K (Jiang et al, 2022) leverages the multi-lingual knowledge base to improve cross-lingual performance on knowledge-related tasks. InfoXLM (Chi et al, 2021) and HiCTL (Wei et al, 2020) encourage bilingual alignment via InfoNCE based contrastive loss.…”

Towards robust natural language and image processing In low-resource scenarios

“…Sequence Tagging Model We implement a BiLSTM-CRF model [200] with the Flair framework [292] to evaluate our data augmentation method on NER and POS tasks. 7 We use a single-layer BiLSTM with hidden state size 512. Dropout layers are applied before and after the BiLSTM layer with dropout rate 0.5.…”

“…Following this, a few recent attempts have been made to enhance multilingual PLMs with Wikipedia or KG triples [7,163,164]. However, due to the structural difference between KG and texts, existing KG based pretraining often relies on extra relation/entity embeddings or additional KG encoders for knowledge enhancement.…”