Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison

Abstract

Word Sense Disambiguation is a longstanding task in Natural Language Processing, lying at the core of human language understanding. However, the evaluation of automatic systems has been problematic, mainly due to the lack of a reliable evaluation framework. In this paper we develop a unified evaluation framework and analyze the performance of various Word Sense Disambiguation systems in a fair setup. The results show that supervised systems clearly outperform knowledge-based models. Among the supervised systems, a linear classifier trained on conventional local features still proves to be a hard baseline to beat. Nonetheless, recent approaches exploiting neural networks on unlabeled corpora achieve promising results, surpassing this hard baseline in most test sets.

Neural Sequence Learning Models for Word Sense Disambiguation

“…It is worth noting that RNN-based architectures outperformed classical supervised approaches (Zhong and Ng, 2010;Iacobacci et al, 2016) when dealing with verbs, which are shown to be highly ambiguous (Raganato et al, 2017). The performance on coarse-grained WSD followed the same trend (Table 2).…”

“…Architecture Details. To set a level playing field with comparison systems on English all-words WSD, we followed Raganato et al (2017) and, for all our models, we used a layer of word embeddings pre-trained 8 on the English ukWaC corpus (Baroni et al, 2009) as initialization, and kept them fixed during the training process. For all architectures we then employed 2 layers of bidirectional LSTM with 2048 hidden units (1024 units per direction).…”

“…Moreover, the field has been explored in depth from different angles by means of extensive empirical studies and evaluation frameworks (Pilehvar and Navigli, 2014;Iacobacci et al, 2016;McCarthy et al, 2016;Raganato et al, 2017).…”

Train-O-Matic: Large-Scale Supervised Word Sense Disambiguation in Multiple Languages without Manual Training Data

“…Gold standard datasets We performed our evaluations using the framework made available by Raganato et al (2017a) on five different allwords datasets, namely: the Senseval-2 (Edmonds and Cotton, 2001), Senseval-3 (Snyder and Palmer, 2004), SemEval-2007(Pradhan et al, 2007, SemEval-2013 and SemEval-2015 (Moro and WSD datasets. We focused on nouns only, given the fact that Wikipedia provides connections between nominal synsets only, and therefore contributes mainly to syntagmatic relations between nouns.…”

EuroSense: Automatic Harvesting of Multilingual Sense Annotations from Parallel Text

“…Given that scaling the manual annotation process becomes practically unfeasible when both lexicographic and encyclopedic knowledge is addressed (Schubert, 2006), recent years have witnessed efforts to produce larger sense-annotated corpora automatically (Moro et al, 2014a;Taghipour and Ng, 2015a;Scozzafava et al, 2015;Raganato et al, 2016). Even though these automatic approaches produce noisier corpora, it has been shown that training on them leads to better supervised and semi-supervised models (Taghipour and Ng, 2015b;Raganato et al, 2016;Yuan et al, 2016;Raganato et al, 2017), as well as to effective embedded representations for senses (Iacobacci et al, 2015;Flekova and Gurevych, 2016).…”