A survey of semantic relatedness evaluation datasets and procedures

Taieb, Mohamed Ali Hadj; Zesch, Torsten; Aouicha, Mohamed Ben

doi:10.1007/s10462-019-09796-3

Cited by 24 publications

(23 citation statements)

References 118 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Polysemy proper adds another dimension of difficulty, since related senses can be perceived to be very similar to one another, and some form of graded relatedness criterion in necessary to properly evaluate model predictions (Erk et al, 2013;Lau et al, 2014). Datasets that capture graded similarity judgements usually do so for word pairs in isolation -often intended to evaluate static word sense embeddings (Taieb et al, 2019), or are conducted on a small number of items (Erk et al, 2013). Notable exceptions are the Word in Context (Wic) dataset by Pilehvar and Camacho-Collados (2019), which contains over 7,000 sentence pairs with an overlapping English word, but was annotated based on a binary classification task.…”

Section: Related Workmentioning

confidence: 99%

Patterns of Polysemy and Homonymy in Contextualised Language Models

Haber¹,

Poesio²

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

One of the central aspects of contextualised language models is that they should be able to distinguish the meaning of lexically ambiguous words by their contexts. In this paper we investigate the extent to which the contextualised embeddings of word forms that display multiplicity of sense reflect traditional distinctions of polysemy and homonymy. To this end, we introduce an extended, human-annotated dataset of graded word sense similarity and co-predication acceptability, and evaluate how well the similarity of embeddings predicts similarity in meaning.Both types of human judgements indicate that the similarity of polysemic interpretations falls in a continuum between identity of meaning and homonymy. However, we also observe significant differences within the similarity ratings of polysemes, forming consistent patterns for different types of polysemic sense alternation. Our dataset thus appears to capture a substantial part of the complexity of lexical ambiguity, and can provide a realistic test bed for contextualised embeddings.Among the tested models, BERT Large shows the strongest correlation with the collected word sense similarity ratings, but struggles to consistently replicate the observed similarity patterns. When clustering ambiguous word forms based on their embeddings, the model displays high confidence in discerning homonyms and some types of polysemic alternations, but consistently fails for others.

show abstract

Section: Related Workmentioning

confidence: 99%

Patterns of Polysemy and Homonymy in Contextualised Language Models

Haber¹,

Poesio²

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

show abstract

“…Several datasets contain human judgments of the similarity or relatedness of (mostly English) word pairs, in isolation (see Taieb et al (2020) for a review). This includes SimLex-999 (Hill et al, 2015), SimVerb-3500 (Gerz et al, 2016), WordSim-353 (Finkelstein et al, 2001), MTurk-771 (Halawi et al, 2012), MEN (Bruni et al, 2014), and more.…”

Section: De-contextualized Word Similarity and Relatednessmentioning

confidence: 99%

RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)

Trott¹,

Bergen²

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Most words are ambiguous-they convey distinct meanings in different contexts-and even the meanings of unambiguous words are context-dependent. Both phenomena present a challenge for NLP. Recently, the advent of contextualized word embeddings has led to success on tasks involving lexical ambiguity, such as Word Sense Disambiguation. However, there are few tasks that directly evaluate how well these embeddings accommodate the continuous, dynamic nature of word meaningparticularly in a way that matches human intuitions. We introduce RAW-C, a dataset of graded, human relatedness judgments for 112 ambiguous words in context (with 672 sentence pairs total), as well as human estimates of sense dominance. The average inter-annotator agreement for the relatedness norms (assessed using a leave-one-annotatorout method) was 0.79. We then show that a measure of cosine distance, computed using contextualized embeddings from BERT and ELMo, correlates with human judgments, but that cosine distance also systematically underestimates how similar humans find uses of the same sense of a word to be, and systematically overestimates how similar humans find uses of different-sense homonyms. Finally, we propose a synthesis between psycholinguistic theories of the mental lexicon and computational models of lexical semantics.

show abstract

“…As mentioned above, semantic similarity is a special case of semantic relatedness [28]. The latter also concerns thematic relations (and, more in general, non-taxonomic relations) and, in the literature, there are several proposals investigating it, by relying on general purpose knowledge resources, such as Wikipedia or WordNet [5], [20], [21], [49]. However, all the approaches in the mentioned literature do not address the intended senses of a pair of concepts in a given application domain, or context, but they consider all the combinations of all possible senses for that pair, and then select the highest values.…”

Section: Related Workmentioning

confidence: 99%

An Enriched Information-Theoretic Definition of Semantic Similarity in a Taxonomy

Formica

Taglino

2021

IEEE Access

View full text Add to dashboard Cite

This paper addresses the notion of semantic similarity between concepts organized according to a taxonomy, based on the well-known information content approach. This approach has been widely experimented in the literature over the years and, in general, outperforms other proposals which do not originate from it. However, it shows some limitations related to the notion of generic sense of a concept. In this paper we illustrate the problem arising by using the traditional approach, and a novel informationtheoretic definition of semantic similarity in a taxonomy is proposed which also takes into account the intended sense of a concept in a given context. This proposal has been applied to some among the most representative state-of-the-art similarity measures based on the information content approach, and the experiment shows that it achieves very high correlation values with human judgment.

show abstract

A survey of semantic relatedness evaluation datasets and procedures

Cited by 24 publications

References 118 publications

Patterns of Polysemy and Homonymy in Contextualised Language Models

Patterns of Polysemy and Homonymy in Contextualised Language Models

RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)

An Enriched Information-Theoretic Definition of Semantic Similarity in a Taxonomy

Contact Info

Product

Resources

About