WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context

Abstract

We present WiC-TSV, a new multi-domain evaluation benchmark for Word Sense Disambiguation. More specifically, we introduce a framework for Target Sense Verification of Words in Context which grounds its uniqueness in the formulation as binary classification task thus being independent of external sense inventories, and the coverage of various domains. This makes the dataset highly flexible for the evaluation of a diverse set of models and systems in and across domains. WiC-TSV provides three different evaluation settings, depending on the input signals provided to the model. We set baseline performance on the dataset using state-of-the-art language models. Experimental results show that even though these models can perform decently on the task, there remains a gap between machine and human performance, especially in outof-domain settings. WiC-TSV data is available at https://competitions.codalab. org/competitions/23683Loïc Vial, Benjamin Lecouteux, and Didier Schwab.2019. Sense vocabulary compression through the semantic knowledge of wordnet for neural word sense disambiguation. In Proceedings of the 10th Global WordNet Conference.

WiC = TSV = WSD: On the Equivalence of Three Semantic Tasks

“…Most English WiC and TSV datasets are based, in whole or in part, on WordNet. If no sense inventory is used for grounding decisions about meaning, the inter-annotator agreement is reported to be only about 80% (Pilehvar and Camacho-Collados, 2019;Breit et al, 2021). For the MCL-WiC dataset, however, annotators consulted other dictionaries, and obtained "almost perfect agreement" (Martelli et al, 2021).…”

“…It is interesting to note that combining such a WordNet lookup with a random back-off on the remaining instances results in correctly solving 76.1% of the WiC instances in the test set, which exceeds the current state-of-the-art of 72.1% (Levine et al, 2020). Breit et al (2021) propose target sense verification (TSV), the task of deciding whether a given word in a given context is used in a given sense. TSV is similar to WiC in that it is also a binary classification task, but only one context is provided.…”

“…We establish the theoretical equivalence of WiC and WSD by specifying reduction algorithms which produce a solution for one problem by applying an algorithm for another. In particular, we employ the target sense verification (TSV) task (Breit et al, 2021) as an intermediate step between WSD and WiC, and specify three reductions: WiC to WSD, WSD to TSV, and TSV to WiC. We formalize the three problems using a common notation, and provide both theoretical and empirical evidence for the correctness of our reductions.…”

Context-Gloss Augmentation for Improving Arabic Target Sense Verification

“…Thus, in the context ( ), WSD aims to determine which of the two senses is the intended meaning of ( ālǧdāwl ). The TSV task is newly proposed in the literature (Breit et al, 2020). It aims to classify a sentence pair with positive or negative.…”

An Analogy based Approach for Solving Target Sense Verification

“…Target Sense Verification (TSV) [19] is a WSD task in which the system is provided with a target word in context on the one hand and a definition and a set of hypernyms of that word on the other hand, and it must decide whether their senses match or not. Consider for instance the context "home is where the heart is", the definition "where you live at a particular time" and the set of hypernyms "residence, abode", all corresponding to the target word home, as shown in Figure 1.…”