Russe'2020: Findings of the First Taxonomy Enrichment Task for the Russian Language

Abstract

This paper describes the results of the first shared task on taxonomy enrichment for the Russian language. The participants were asked to extend an existing taxonomy with previously unseen words: for each new word their systems should provide a ranked list of possible (candidate) hypernyms. In comparison to the previous tasks for other languages, our competition has a more realistic task setting: new words were provided without definitions. Instead, we provided a textual corpus where these new terms occurred. For this evaluation campaign, we developed a new evaluation dataset based on unpublished RuWordNet data. The shared task features two tracks: “nouns” and “verbs”. 16 teams participated in the task demonstrating high results with more than a half of them outperforming the provided baseline.

Studying Taxonomy Enrichment on Diachronic WordNet Versions

“…Thus, following RUSSE-2020 shared task (Nikishina et al, 2020) we stick to a more realistic scenario when we have no definitions of new words, but only examples of their usage. The organisers of the shared task provided a baseline and training and evaluation datasets based on RuWordNet (Loukachevitch et al, 2016).…”

“…We show how it can be transferred to other wordnets or taxonomies with timestamped releases for Russian. We create an analogous version to English extending the dataset by Nikishina et al (2020) based on RuWordNet. The original dataset does not include short words (< 4 symbols), diminutives, named entities and other constraints described in the shared task paper.…”

“…On the other hand, the distributional hypothesis states that words that occur in similar context tend to have similar meanings (Harris, 1954). Our approach is built on the top of the baseline method by Nikishina et al (2020). In this method, top k = 10 nearest neighbours of the input word are taken from the pre-trained embedding model (according to the above considerations they should be co-hyponyms).…”

“…At the same time, we should bear in mind that guessing a direct hypernym or a second-order hypernym are equally good options. Therefore, following Nikishina et al (2020), we evaluate our models with modified MAP. It transforms a list of gold standard hypernyms into a list of connected components.…”

“…• We create datasets for studying diachronic evolution of wordnets for English and Russian, extending the monolingual setup of the RUSSE'2020 shared task (Nikishina et al, 2020) with a larger Russian dataset and similar English versions. 2 • We present several new simple yet efficient methods for taxonomy enrichment.…”