Capturing Evolution in Word Usage: Just Add More Clusters?

Abstract

The way the words are used evolves through time, mirroring cultural or technological evolution of society. Semantic change detection is the task of detecting and analysing word evolution in textual data, even in short periods of time. In this paper we focus on a new set of methods relying on contextualised embeddings, a type of semantic modelling that revolutionised the NLP field recently. We leverage the ability of the transformer-based BERT model to generate contextualised embeddings capable of detecting semantic change of words across time. Several approaches are compared in a common setting in order to establish strengths and weaknesses for each of them. We also propose several ideas for improvements, managing to drastically improve the performance of existing approaches.

RuSemShift: a dataset of historical lexical semantic change in Russian

“…Jensen-Shannon divergence (JSD). In this measure, influenced by Dubossarsky et al (2015), Martinc et al (2020) and , word usage matrices from two time periods are first stacked into one matrix. Then, we standardize the vectors and obtain word usage clusters of token embeddings using the Affinity Propagation clustering algorithm (Frey and Dueck, 2007).…”

Analysing Lexical Semantic Change with Contextualised Word Representations

“…There also exist clustering methods that select the optimal K automatically, e.g. DBSCAN or Affinity Propagation (Martinc et al, 2020). They nevertheless require method-specific parameter choices which indirectly determine the number of clusters.…”

ELMo and BERT in Semantic Change Detection for Russian

“…This provides new opportunities for diachronic analysis: for example, it is possible to group similar token representations and measure a diversity of such representations, while predefined number of senses is not strictly necessary. Thus, currently there is an increased interest in the topic of language change detection using contextualized word embeddings [9,10,14,21,27,28].…”

“…[27] used averaged time-specific BERT representations and calculated cosine distance between averaged vectors of two time periods as a measure of semantic change. [28] tested Affinity Propagation algorithm for usage clusterization and showed that it is consistently better than k-Means. Finally, [21] applied approaches similar to [10], but also analyzing ELMo models and adding cosine similarity of average vectors as a measure.…”