Hypernyms under Siege: Linguistically-motivated Artillery for Hypernymy Detection

Abstract

The fundamental role of hypernymy in NLP has motivated the development of many methods for the automatic identification of this relation, most of which rely on word distribution. We investigate an extensive number of such unsupervised measures, using several distributional semantic models that differ by context type and feature weighting. We analyze the performance of the different methods based on their linguistic motivation. Comparison to the state-of-the-art supervised methods shows that while supervised methods generally outperform the unsupervised ones, the former are sensitive to the distribution of training instances, hurting their reliability. Being based on general linguistic hypotheses and independent from training data, unsupervised measures are more robust, and therefore are still useful artillery for hypernymy detection.

SemEval-2018 Task 9: Hypernym Discovery

“…We generated a list of candidate hypernyms for each target word, and then employed unsupervised hypernymy detection measures to decide whether a hypernymy relation holds. We used the opensource code by Shwartz et al (2017). 15 Our baseline starts by creating a distributional semantic model (DSM) for each domain/language (English, Spanish, Italian, Music and Medical).…”

“…Similarly to the hyponym selection step (Section 4.1.3), all the terms with frequency of at least 3 occurrences in the source corpus are considered as valid targets. For the context words, instead, we required a minimum of 100 occurrences, as in Shwartz et al (2017). To generate candidates, we took the 50 most similar terms for each target word via cosine similarity in the DSM.…”

“…16 Finally, we tuned the thresholds for the above measures by maximizing the average of the performance metrics on the training set, separately for each subtask and measure. 14 We used the open-source code available at https:// bitbucket.org/luisespinosa/taxoembed 15 https://github.com/vered1986/ UnsupervisedHypernymy 16 Following the conclusions from Shwartz et al (2017), we set the hyper-parameters to: SLQS: median, PLMI, N = 100 and APSyn: N = 500. Table 3 shows a summary of all participant systems, displaying their main features with respect to supervison and external resources used, if any.…”

“…Criticisms to this experimental setting point out that supervised systems tend to benefit from the inherent modeling of the datasets in the hypernym detection task, leading to lexical memorization phenomena (Levy et al, 2015;Santus et al, 2016a;Shwartz et al, 2017). In this respect, recent work has attempted to alleviate this issue by including a graded scale for evaluating the degree of hypernymy on a given pair (Vulić et al, 2017).…”

Dual Tensor Model for Detecting Asymmetric Lexico-Semantic Relations

“…All distributional models we evaluate achieve poorer performance on meronymy than hypernymy detection, especially considering that WN-Me is a balanced dataset, whereas HypeNet is heavily skewed towards negative instances. Shwartz et al (2017) propose ranking as an alternative evaluation setting for hypernymy detection. The goal is to rank positive relation pairs higher than negative ones.…”