CMCE at SemEval-2020 Task 1: Clustering on Manifolds of Contextualized Embeddings to Detect Historical Meaning Shifts

Abstract

This paper describes the system Clustering on Manifolds of Contextualized Embeddings (CMCE) submitted to the SemEval-2020 Task 1 on Unsupervised Lexical Semantic Change Detection. Subtask 1 asks to identify whether or not a word gained/lost a sense across two time periods. Subtask 2 is about computing a ranking of words according to the amount of change their senses underwent. Our system uses contextualized word embeddings from MBERT, whose dimensionality we reduce with an autoencoder and the UMAP algorithm, to be able to use a wider array of clustering algorithms that can automatically determine the number of clusters. We use Hierarchical Density Based Clustering (HDBSCAN) and compare it to Gaussian Mixture Models (GMMs) and other clustering algorithms. Remarkably, with only 10 dimensional MBERT embeddings (reduced from the original size of 768), our submitted model performs best on subtask 1 for English and ranks third in subtask 2 for English. In addition to describing our system, we discuss our hyperparameter configurations and examine why our system lags behind for the other languages involved in the shared task (German, Swedish, Latin

SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection

“…NLPCR (Rother et al, 2020) The team uses multilingual contextualized word embeddings (Devlin et al, 2019) to represent a word's meaning. They then reduce the embedding dimensionality with either autoencoder or UMAP, and cluster the resulting representation with either GMM or HDBSCAN.…”

Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slang

“…Dimensionality Reduction: To the best of our knowledge, only one previous semantic change detection approach (Rother et al, 2020) has incorporated dimensionality reduction, more specifically UMAP (McInnes et al, 2018). As the Euclidean distances in the UMAP-reduced space are very sensitive to hyperparameters and it does not retain an interpretable notion of absolute distances, it might be unsuitable for pure distance-based metrics like APD, and we therefore also experiment with PCA.…”

Forecasting COVID-19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts

“…UMAP (Uniform Manifold Approximation and Projection for Dimension Reduction) is used as in Rother et al (2020) to lower the dimensionality of the embedding space. UMAP is appropriate for this task since it preserves global structure better than other manifold learning dimensionality reduction methods such as t-SNE (McInnes et al, 2018) (McConville et al, 2021.…”

“…In a similar vein, Aharoni and Goldberg (2020) showed that the domain type of a particular text could be identified using the clustering of sentence-level representations. Finally, Rother et al (2020) showed that clusters of contextualised embeddings could detect meaning shifts in words. The success of these papers motivates our use of high-density clusters of sentencelevel representations.…”