What Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition

Abstract

Contextualized word embeddings have boosted many NLP tasks compared with traditional static word embeddings. However, the word with a specific sense may have different contextualized embeddings due to its various contexts. To further investigate what contextualized word embeddings capture, this paper analyzes whether they can indicate the corresponding sense definitions and proposes a general framework that is capable of explaining word meanings given contextualized word embeddings for better interpretation. The experiments show that both ELMo and BERT embeddings can be well interpreted via a readable textual form, and the findings may benefit the research community for a better understanding of what the embeddings capture 1 .

Generationary or “How We Went beyond Word Sense Inventories and Learned to Gloss”

“…We rank definitions by applying the MBRR plus cosine similarity strategy described in Section 3.2.3. Chang and Chen (2019) achieves, in most cases, the highest recovery rate. However, with k = 1, which is the most realistic case, Gen-CHA S outperforms the competitor by 4.6 points when macro-averaging on senses, i.e.…”

“…Recent approaches have explored the use of large-scale pre-trained models to score definitions with respect to a usage context. For example, Chang and Chen (2019) proposed to recast DM as a definition ranking problem. A similar idea was applied in WSD by Huang et al (2019), leading to state-of-the-art results.…”

“…CHA (Chang and Chen, 2019) is an online dataset 4 SEM is a dataset built by exploiting the Sem-Cor corpus (Miller et al, 1993) -which is manually tagged with WordNet senses -to associate sentence-level contexts with definitions. We filter out NER-like sense annotations (e.g.…”

“…We use CHA and SEM since they were employed by state-of-the-art approaches to DM (Chang and Chen, 2019) and WSD (Huang et al, 2019). With UNI, instead, we bring together diverse sense inventories to create a dataset that is less dependent on the idiosyncrasies of each of its sources.…”

“…For each evaluation dataset D we test two Generationary models: one trained on the corresponding train split (Gen-D), and one trained on UNI (Gen-UNI). 8 We compare against (i) a random baseline which predicts, for each test item, a random definition taken from the same test set; (ii) the model of Ishiwatari et al (2019), which we have re-trained on the same data as Generationary (Ishiwatari-D), and (iii) the state-of-the-art approach of Chang and Chen (2019, Chang). On HEI, which has no training split, we only evaluate Gen-UNI and the random baseline, since Ishiwatari-UNI generates strings consisting of mostly unknown word placeholders (<unk>), and Chang and Chen (2019) cannot handle multi-word targets.…”