Lexicon Induction for Spoken Rusyn – Challenges and Results

Abstract

This paper reports on challenges and results in developing NLP resources for spoken Rusyn. Being a Slavic minority language, Rusyn does not have any resources to make use of. We propose to build a morphosyntactic dictionary for Rusyn, combining existing resources from the etymologically close Slavic languages Russian, Ukrainian, Slovak, and Polish. We adapt these resources to Rusyn by using vowel-sensitive Levenshtein distance, hand-written language-specific transformation rules, and combinations of the two. Compared to an exact match baseline, we increase the coverage of the resulting morphological dictionary by up to 77.4% relative (42.9% absolute), which results in a tagging recall increased by 11.6% relative (9.1% absolute). Our research confirms and expands the results of previous studies showing the efficiency of using NLP resources from neighboring languages for low-resourced languages.

Multi-source morphosyntactic tagging for spoken Rusyn

“…In Rabus and Scherrer (2017), we describe the automatic induction of morphosyntactic lexicons for Rusyn. In a nutshell, we match Rusyn words extracted from RUE1 and RUE2 with source language words extracted from the Polish, Slovak, Ukrainian and Russian MULTEXT-East lexicons as well as the morphological dictionary of UGtag 9 (Kotsyba et al, 2011), using vowel-sensitive Levenshtein distance, hand-written rules, and a combination of both.…”

Neural morphosyntactic tagging for Rusyn

“…In contrast, tagging performance on the Rusyn test set decreases substantially despite the lower OOV rate. In Rabus and Scherrer (2017), we have found that a large proportion of Rusyn word forms can be matched with source language word forms using hand-crafted correspondence rules or vowel-sensitive Levenshtein distance. Here, we use the latter approach to artificially create Rusyn word embeddings for the out-of-vocabulary words: each Rusyn word form of the test set that does not occur in the Panslav5 file is associated with the most similar forms in Panslav5, and its embedding vector is computed by averaging the vectors of the found similar forms.…”

“…In our previous work on Rusyn morphosyntactic tagging, we followed a twofold approach. First, we built a morphosyntactic dictionary of Rusyn, applying bilingual lexicon induction techniques on corpora from the East Slavic languages Russian and Ukrainian and on cyrillicized corpora from the West Slavic languages Slovak and Polish (Rabus and Scherrer 2017). Word forms were matched by vowel-sensitive Levenshtein distance, manually written transformation rules and combinations of both.…”