SenseCluster at SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection

Abstract

We (Team Skurt) propose a simple method to detect lexical semantic change by clustering contextualized embeddings produced by XLM-R, using K-Means++. The basic idea is that contextualized embeddings that encode the same sense are located in close proximity in the embedding space. Our approach is both simple and generic, but yet performs relatively well in both sub-tasks of SemEval-2020 Task 1. We hypothesize that the main shortcoming of our method lies in the simplicity of the clustering method used.

Contextualized embeddings for semantic change detection: Lessons learned

“…Overall, these methods can be considered typical representatives of using contextualized word embeddings for the task of semantic change detection: they boil down to directly comparing token embeddings of the target word in two periods; see (Martinc et al, 2020a) for a similar technique. Another possible approach (which we hope to analyze in the future) is clustering token embeddings into groups loosely corresponding to word senses and then comparing their time-specific distributions (Martinc et al, 2020b;Cuba Gyllensten et al, 2020;Giulianelli et al, 2020).…”

What is Done is Done: an Incremental Approach to Semantic Shift Detection

“…A number of unsupervised approaches based on contextual embeddings are proposed to sidestep the need of lexicographic resources (Schlechtweg et al, 2020;Tahmasebi et al, 2021). In general, these kinds of approaches follow a three-step scheme: i) extraction of embeddings for each occurrence of a target word from a contextual model such as BERT (Hu et al, 2019;Martinc et al, 2020a), ELMo (Kutuzov and Giulianelli, 2020;Rodina et al, 2020), or XLM-R (Cuba Gyllensten et al, 2020;Rother et al, 2020); ii) aggregation of the embeddings with a clustering algorithm like K-Means (Giulianelli et al, 2020;Cuba Gyllensten et al, 2020), Affinity Propagation (Martinc et al, 2020a;Kutuzov and Giulianelli, 2020), or DBSCAN (Rother et al, 2020;Karnysheva and Schwarz, 2020); iii) comparison of the vector distribution over clusters according to time by using a semantic distance measure, like Jensen-Shannon divergence (Martinc et al, 2020a), Entropy Difference (Giulianelli et al, 2020), or Wasserstein Distance (Montariol et al, 2021). The main limitation of applying clustering to word embeddings is the scalability issues about memory consumption and time.…”

“…The use of contextual embedding techniques is receiving more and more attention in the field of semantic shift detection. In particular, pre-trained models like BERT (Hu et al, 2019;Martinc et al, 2020a), ELMo (Kutuzov and Giulianelli, 2020;Rodina et al, 2020), and XLM-R (Cuba Gyllensten et al, 2020;Rother et al, 2020), are being proposed as promising solutions to capture the different meanings of a target word according to the different contexts in which the word appears throughout a considered diachronic corpus. Such solutions generally employ clustering techniques to aggregate embeddings of a specific word into clusters (Martinc et al, 2020a;Karnysheva and Schwarz, 2020).…”

SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection

“…Skurt (Gyllensten et al, 2020) The team uses pretrained cross-lingual contextualized embedding model, XLM-R (Conneau et al, 2019), which enables them to use the same model for all languages. For each target word they generated contextual representations (token representations) from the two corpora, and cluster them using K-Means++ with a fixed number of clusters (8).…”