SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation

Abstract

Contextual representations of words derived by neural language models have proven to effectively encode the subtle distinctions that might occur between different meanings of the same word. However, these representations are not tied to a semantic network, hence they leave the word meanings implicit and thereby neglect the information that can be derived from the knowledge base itself. In this paper, we propose SensEmBERT, a knowledge-based approach that brings together the expressive power of language modelling and the vast amount of knowledge contained in a semantic network to produce high-quality latent semantic representations of word meanings in multiple languages. Our vectors lie in a space comparable with that of contextualized word embeddings, thus allowing a word occurrence to be easily linked to its meaning by applying a simple nearest neighbour approach.We show that, whilst not relying on manual semantic annotations, SensEmBERT is able to either achieve or surpass state-of-the-art results attained by most of the supervised neural approaches on the English Word Sense Disambiguation task. When scaling to other languages, our representations prove to be equally effective as their English counterpart and outperform the existing state of the art on all the Word Sense Disambiguation multilingual datasets. The embeddings are released in five different languages at http://sensembert.org.

Adapting BERT for Word Sense Disambiguation with Gloss Selection Objective and Example Sentences

“…Besides deep learning-based approach, Loureiro and Jorge (2019) and Scarlini et al (2020) construct sense embeddings using the contextual embeddings from BERT. The former generates sense embeddings by averaging the contextual embeddings of sense-annotated tokens taken from Sem-Cor while the latter constructs sense embeddings by concatenating the contextual embeddings of Ba-belNet definitions with the contextual embeddings of Wikipedia contexts.…”

“…The former generates sense embeddings by averaging the contextual embeddings of sense-annotated tokens taken from Sem-Cor while the latter constructs sense embeddings by concatenating the contextual embeddings of Ba-belNet definitions with the contextual embeddings of Wikipedia contexts. For WSD, both approaches make use of the constructed sense embeddings in nearest neighbor classification (kNN), in which the simple 1-nearest neighbor approach from Scarlini et al (2020) showed substantial improvement over the nominal category of the English all-words WSD benchmark datasets.…”

Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations

“…A further step towards semantic representations involves modelling individual word senses as vectors which are explicitly linked to a knowledge resource. Early approaches to sense embeddings adapted existing work to project words and word senses onto a shared distributional space (Iacobacci et al, 2015;Iacobacci and Navigli, 2019), while more recent studies exploited the inner states of pretrained language models (Loureiro and Jorge, 2019;Scarlini et al, 2020a). Instead of modelling language-specific units like words or senses, NASARI (Camacho-Collados et al, 2016) represents language-independent concepts using sparse lexical vectors.…”

“…As a result, contextualized word representations have garnered attention (Melamud et al, 2016), enjoying great success in the form of pretrained language models like BERT (Devlin et al, 2019) or XLM (Conneau and Lample, 2019). At the same time, modelling techniques for individual word senses, concepts and named entities have also gained traction (Camacho-Collados et al, 2016;Scarlini et al, 2020a), though their integration into downstream NLP applications is still subject of ongoing investigations (Li and Jurafsky, 2015;.…”

CSI: A Coarse Sense Inventory for 85% Word Sense Disambiguation

“…It is a key task in Natural Language Processing (Navigli 2018), providing semantic information that is potentially beneficial for downstream applications, such as information extraction (Delli Bovi, Espinosa Anke, and Navigli 2015) and machine translation (Pu et al 2018). While much effort has been devoted to building new algorithms or data (Pasini and Navigli 2018;Scarlini, Pasini, and Navigli 2019) for this task, state-ofthe-art systems have yet to break the 80% accuracy ceiling on standard WSD benchmark datasets (Raganato, Delli Bovi, and Navigli 2017;Bevilacqua and Navigli 2019;Vial, Lecouteux, and Schwab 2019;Scarlini, Pasini, and Navigli 2020), showing that the WSD task is far from being solved. Following the literature in the field (Hovy et al 2006;Palmer, Dang, and Fellbaum 2007;Navigli, Litkowski, and Hargraves 2007), we argue that the reason for this unsatisfactory performance does not lie solely in the complexity of the task but also in the fine granularity of the sense inventory adopted, i.e., WordNet (Fellbaum 1998).…”