DeepMistake: Which Senses are Hard to Distinguish for a Word­in­Context Model

Arefyev, Nikolay; Fedoseev, M. V.; Protasov, Vitaly; Panchenko, Alexander; Homskiy, Daniil; Davletov, Adis

doi:10.28995/2075-7182-2021-20-16-30

Cited by 4 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For comparability, we use the same model for all the languages. We choose XLM-R (Conneau et al, 2020), a multilingual Transformer-based masked language model which has already been successfully applied to the semantic change detection task (Arefyev and Zhikov, 2020;Arefyev et al, 2021). Although it covers the full linguistic diversity of our data, we additionally fine-tune XLM-R on monolingual diachronic corpora.…”

Section: Introductionmentioning

confidence: 99%

Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change

Giulianelli¹,

Kutuzov²,

Pivovarova³

2022

Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change

View full text Add to dashboard Cite

Morphological and syntactic changes in word usage-as captured, e.g., by grammatical profiles-have been shown to be good predictors of a word's meaning change. In this work, we explore whether large pre-trained contextualised language models, a common tool for lexical semantic change detection, are sensitive to such morphosyntactic changes. To this end, we first compare the performance of grammatical profiles against that of a multilingual neural language model (XLM-R) on 10 datasets, covering 7 languages, and then combine the two approaches in ensembles to assess their complementarity. Our results show that ensembling grammatical profiles with XLM-R improves semantic change detection performance for most datasets and languages. This indicates that language models do not fully cover the finegrained morphological and syntactic signals that are explicitly represented in grammatical profiles.An interesting exception are the test sets where the time spans under analysis are much longer than the time gap between them (for example, century-long spans with a one-year gap between them). Morphosyntactic change is slow so grammatical profiles do not detect in such cases. In contrast, language models, thanks to their access to lexical information, are able to detect fast topical changes.

show abstract

Section: Introductionmentioning

confidence: 99%

Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change

Giulianelli¹,

Kutuzov²,

Pivovarova³

2022

Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change

View full text Add to dashboard Cite

show abstract

“…In(Arefyev et al, 2021) it was observed that taking more than 100 pairs does not significantly improve the results, though this was observed for a different model.…”

mentioning

confidence: 96%

GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish

Rachinskiy¹,

Arefyev²

2022

Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change

Self Cite

View full text Add to dashboard Cite

The contextualized embeddings obtained from neural networks pre-trained as Language Models (LM) or Masked Language Models (MLM) are not well suitable for solving the Lexical Semantic Change Detection (LSCD) task because they are more sensitive to changes in word forms rather than word meaning, a property previously known as the word form bias or orthographic bias (Laicher et al., 2021). Unlike many other NLP tasks, it is also not obvious how to fine-tune such models for LSCD. In order to conclude if there are any differences between senses of a particular word in two corpora, a human annotator or a system shall analyze many examples containing this word from both corpora. This makes annotation of LSCD datasets very labour-consuming. The existing LSCD datasets contain up to 100 words that are labeled according to their semantic change, which is hardly enough for fine-tuning.To solve these problems we fine-tune the XLM-R MLM (Conneau et al., 2020) as part of a gloss-based WSD system on a large WSD dataset in English. Then we employ zero-shot cross-lingual transferability of XLM-R to build the contextualized embeddings for examples in Spanish. In order to obtain the graded change score for each word, we calculate the average distance between our improved contextualized embeddings of its old and new occurrences. For the binary change detection subtask, we apply thresholding to the same scores.Our solution has shown the best results among all other participants in all subtasks except for the optional sense gain detection subtask.

show abstract

Sense through time: diachronic word sense annotations for word sense induction and Lexical Semantic Change Detection

Schlechtweg,

Zamora-Reina,

Bravo-Marquez

et al. 2024

Lang Resources & Evaluation

View full text Add to dashboard Cite

There has been extensive work on human word sense annotation, i.e., manually labeling word uses in natural texts according to their senses. Such labels were primarily created for the tasks of Word Sense Disambiguation (WSD) and Word Sense Induction (WSI). However, almost all datasets annotated with word senses are synchronic datasets, i.e., contain texts created in a relatively short period of time and often do not provide the creation date of the texts. This ignores possible applications in diachronic-historic settings, where the aim is to induce or disambiguate historical word senses or changes in senses across time. To facilitate investigations into historical WSD and WSI and to establish connections with the task of Lexical Semantic Change Detection (LSCD), there is a crucial need for historical word sense-annotated data. Hence, we created a new reliable diachronic WSD/WSI dataset ‘DWUG DE Sense’. We describe the preparation and annotation and analyze central statistics. We then describe a thorough evaluation of different prediction systems for jointly solving both WSI and LSCD tasks. All our systems are based on a state-of-the-art architecture that combines Word-in-Context models and graph clustering techniques with different hyperparameter settings. Our findings reveal that using the WSI task as optimization criterion yields better results for both tasks even when the LSCD task is the focal point of optimization. This underscores that although both tasks are related, WSI seems to be more general and able to incorporate the LSCD task.

show abstract

DeepMistake: Which Senses are Hard to Distinguish for a WordinContext Model

Cited by 4 publications

References 6 publications

Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change

Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change

GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish

Sense through time: diachronic word sense annotations for word sense induction and Lexical Semantic Change Detection

Contact Info

Product

Resources

About

DeepMistake: Which Senses are Hard to Distinguish for a Word­in­Context Model

Cited by 4 publications

References 6 publications

Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change

Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change

GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish

Sense through time: diachronic word sense annotations for word sense induction and Lexical Semantic Change Detection

Contact Info

Product

Resources

About

DeepMistake: Which Senses are Hard to Distinguish for a WordinContext Model