GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish

Abstract

The contextualized embeddings obtained from neural networks pre-trained as Language Models (LM) or Masked Language Models (MLM) are not well suitable for solving the Lexical Semantic Change Detection (LSCD) task because they are more sensitive to changes in word forms rather than word meaning, a property previously known as the word form bias or orthographic bias (Laicher et al., 2021). Unlike many other NLP tasks, it is also not obvious how to fine-tune such models for LSCD. In order to conclude if there are any differences between senses of a particular word in two corpora, a human annotator or a system shall analyze many examples containing this word from both corpora. This makes annotation of LSCD datasets very labour-consuming. The existing LSCD datasets contain up to 100 words that are labeled according to their semantic change, which is hardly enough for fine-tuning.To solve these problems we fine-tune the XLM-R MLM (Conneau et al., 2020) as part of a gloss-based WSD system on a large WSD dataset in English. Then we employ zero-shot cross-lingual transferability of XLM-R to build the contextualized embeddings for examples in Spanish. In order to obtain the graded change score for each word, we calculate the average distance between our improved contextualized embeddings of its old and new occurrences. For the binary change detection subtask, we apply thresholding to the same scores.Our solution has shown the best results among all other participants in all subtasks except for the optional sense gain detection subtask.

Methodological Insights in Detecting Subtle Semantic Shifts with Contextualized and Static Language Models

“…Our experiments confirm the intuition from Schlechtweg et al (2020) that information learned in pretraining provides such a strong signal that contexts reflecting a new sense of a target token do not impact its representation. (2) The best-performing LSCD system of the Russian and Spanish shared task (Rachinskiy and Arefyev, 2022) did not perform best on our subtle shift task nor our evaluation on the English subtask of Se-mEval 2020. (3) We propose an alternative method for contextualized models relying on masked token prediction rather than representation comparison inspired by Arefyev and Zhikov (2020).…”

“…The pretrained (and domain-adapted) model are also further fine-tuned on a specific task to incorporate information relevant to semantic change. So far, a method employing fine-tuning for Word Sense Disambiguation is the only method for contextualized models that has been shown to outperform static models (Rachinskiy and Arefyev, 2022) on two recent shared tasks on LSCD in Russian (Kutuzov and Pivovarova, 2021) and Spanish (Zamora-Reina et al, 2022), respectively. The intuition behind this approach is that the fine-tuning step foregrounds information about word senses rather than word forms.…”

Understanding Computational Models of Semantic Change: New Insights from the Speech Community

“…Token-level models provide contextualized embeddings that can be pooled into time-specific representations or used in clustering (Martinc et al, 2020;Giulianelli et al, 2020). Other deep neural architectures use cross-lingual transfer (Rachinskiy and Arefyev, 2022) or train on temporal data (Rosin and Radinsky, 2022). Similar methods are used to compare word meanings across online communities (Del Tredici and Fernández, 2017), text types (Fišer and Ljubešić, 2018), dialect regions (Kulkarni et al, 2016), and languages (Uban et al, 2019).…”

Towards Detecting Lexical Change of Hate Speech in Historical Data

“…Hamilton et al (2016), outperformed other approaches, including those using BERT-based models (Kutuzov and Giulianelli, 2020). More recently, several methods based on contextualized models have shown greater success, either by extracting representations from a Transformer-based model fine-tuned on Word Sense Disambiguation (Rachinskiy and Arefyev, 2022), or relying on the most probable substitutes for masked target terms (Card, 2023). In our study, we adopt a method loosely based on the latter approach, which we will elaborate on in Section 4.2.…”