EViLBERT: Learning Task-Agnostic Multimodal Sense Embeddings

Abstract

<jats:p>The problem of grounding language in vision is increasingly attracting scholarly efforts. As of now, however, most of the approaches have been limited to word embeddings, which are not capable of handling polysemous words. This is mainly due to the limited coverage of the available semantically-annotated datasets, hence forcing research to rely on alternative technologies (i.e., image search engines). To address this issue, we introduce EViLBERT, an approach which is able to perform image classification over an open set of concepts, both concrete and non-concrete. Our approach is based on the recently introduced Vision-Language Pretraining (VLP) model, and builds upon a manually-annotated dataset of concept-image pairs. We use our technique to clean up the image-to-concept mapping that is provided within a multilingual knowledge base, resulting in over 258,000 images associated with 42,500 concepts. We show that our VLP-based model can be used to create multimodal sense embeddings starting from our automatically-created dataset. In turn, we also show that these multimodal embeddings improve the performance of a Word Sense Disambiguation architecture over a strong unimodal baseline. We release code, dataset and embeddings at http://babelpic.org.</jats:p>

Sememe Prediction for BabelNet Synsets using Multilingual and Multimodal Information

“…The image encoder is used to capture the semantic information contained in the images in a BabelNet synset. Previous studies have shown that images can help learn better semantic representations for concepts and entities (Xie et al, 2017a;Calabrese et al, 2020). We believe that images are also beneficial to SPBS.…”

“…It has been utilized in multiple NLP tasks (Navigli et al, 2021), especially the cross-lingual or multilingual tasks, such as multilingual word sense disambiguation (Navigli and Ponzetto, 2012b), cross-lingual lexical entailment (Vyas and Carpuat, 2016) and cross-lingual AMR parsing (Blloshmi et al, 2020). Most of these studies regard BabelNet as a large multilingual sense inventory and utilize the multilingual synonyms and glosses in BabelNet synsets, and some studies also use images in it, e.g., Calabrese et al (2020) learn multimodal sense embeddings with the concepts and images in BabelNet.…”

Recent Trends in Word Sense Disambiguation: A Survey

“…We note that different kinds of knowledge are orthogonal to each other and can be exploited in conjunction. For example, token classification models benefit from the logits-adjacency matrix multiplication , binary cross-entropy training , translation-based refinement [Luan et al, 2020] and visual information [Calabrese et al, 2020a].…”

“…In a different direction, Calabrese et al [2020a] leverage images from the BabelPic dataset [Calabrese et al, 2020b] to build multimodal gloss vectors, which are shown to be stronger than text-only vectors when used to initialize the weights of the classification matrix ( in Eq. 1).…”

SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC)

“…using LSTMs (Melamud et al, 2016) or the Transformer architecture (Devlin et al, 2019;Conneau et al, 2020), and are capable of representing words based on the context in which they occur. Contextualized representations have also been used to obtain effective sense embeddings (Loureiro and Jorge, 2019;Scarlini et al, 2020a,b;Calabrese et al, 2020).…”