A comparative evaluation and analysis of three generations of Distributional Semantic Models

Abstract

Distributional semantics has deeply changed in the last decades. First, predict models stole the thunder from traditional count ones, and more recently both of them were replaced in many NLP applications by contextualized vectors produced by neural language models. Although an extensive body of research has been devoted to Distributional Semantic Model (DSM) evaluation, we still lack a thorough comparison with respect to tested models, semantic tasks, and benchmark datasets. Moreover, previous work has mostly focused on task-driven evaluation, instead of exploring the differences between the way models represent the lexical semantic space. In this paper, we perform a large-scale evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. First of all, we investigate the performance of embeddings in several semantic tasks, carrying out an in-depth statistical analysis to identify the major factors influencing the behavior of DSMs. The results show that (i) the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous and (ii) static DSMs surpass BERT representations in most out-of-context semantic tasks and datasets. Furthermore, we borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models. RSA reveals important differences related to the frequency and part-of-speech of lexical items.

Patterns in CAOSS: Distributed representations predict variation in relational interpretations for familiar and novel compound words

“…We extracted all semantic vectors from a stateof-the art model for representing lexical semantics (Lenci, Sahlgren, Jeuniaux, Gyllensten, & Miliani, 2021), which was identified as the best-performing distributional semantic model in the systematic evaluation study by Baroni, Dinu, and Kruszewski (2014): A cbow variant of the word2vec algorithm (Mikolov, Chen, Corrado, & Dean, 2013), aimed at predicting each target word in the corpus from the 5 words to its left and its right (i.e., window size 5), negative sampling with k = 10, and subsampling with t = 1e −5 . The model was trained on an English ∼ 2.8 billion word corpus (a concatenation of the ukWaC corpus, an English Wikipedia dump, and the British National Corpus).…”

Similar-sounding words flesh out fuzzy meanings

“…The bulk of the English lexicon is low in frequency (e.g., Webb & Rodgers, 2009), yet people struggle to process those low-frequency words (e.g., Monsell et al, 1989), as do state of the art computational models (e.g., Lenci et al, 2022). We have provided evidence that English speakers compensate for a lack of experience with words by recruiting similar-sounding words with related meanings.…”

“…Unlike previous lexical decision tasks that investigated words that overlap with other words in both form and meaning (e.g., Bergen, 2004;Pastizzo & Feldman, 2009), this semantic relatedness task requires word meaning access. Indeed, semantic relatedness tasks are often used to evaluate the performance of DSMs (e.g., Baroni, Dinu, & Kruszewski, 2014;Mandera et al, 2017;Lenci et al, 2022). Moreover, unlike those masked priming studies, participants in this experiment are never presented with the similar-sounding attractor words (e.g., avoid).…”

“…These models are fascinating, but their emphasis on context may detract from the content of the lexicon. Indeed, Lenci, Sahlgren, Jeuniaux, Gyllensten, and Miliani (2022) found that static DSMs outperform contextualized DSMs (namely BERT; Devlin, Chang, Lee, & Toutanova, 2018) in out-of-context tasks, such as identifying synonyms or gauging semantic relatedness. More to the point, Lenci et al (2022) found that DSMs continue to struggle with low-frequency words: Vectors of low-frequency words are idiosyncratic, with little in common across models.…”

“…(However, that model produces quite a few negative cosine values, and the p-value drops below 0.2 only after setting those negative values to zero, as advised byGünther et al, 2015, because negative cosines in DSMs cannot be reliably interpreted. This may exemplify how DSMs struggle with low-frequency words, as discussed above; e.g.,Sahlgren & Lenci, 2016;Lenci et al, 2022. ) To ensure that the sentences containing the arbitrary and non-arbitrary target words are equally semantically related to the corresponding cue sentences, we computed the conceptual overlap of those pairs of sentences using the coherence() function in the LSAfun package.…”