Leveraging Inflection Tables for Stemming and Lemmatization.

Abstract

We present several methods for stemming and lemmatization based on discriminative string transduction. We exploit the paradigmatic regularity of semi-structured inflection tables to identify stems in an unsupervised manner with over 85% accuracy. Experiments on English, Dutch and German show that our stemmers substantially outperform Snowball and Morfessor, and approach the accuracy of a supervised model. Furthermore, the generated stems are more consistent than those annotated by experts. Our direct lemmatization model is more accurate than Morfette and Lemming on most datasets. Finally, we test our methods on the data from the shared task on morphological reinflection.

Context Sensitive Lemmatization Using Two Successive Bidirectional Gated Recurrent Networks

“…Again, the later methods can be further classified depending on whether context of the current word is considered or not. Lemmatization without context (Cotterell et al, 2016;Nicolai and Kondrak, 2016) is closer to stemming and not the focus of the present work. It is noteworthy here that the supervised lemmatization methods do not try to classify the lemma of a given word form as it is infeasible due to having a large number of lemmas in a language.…”

“…Efforts on developing lemmatizers can be divided into two principle categories (i) rule/heuristics based approaches (Koskenniemi, 1984;Plisson et al, 2004) which are usually not portable to different languages and (ii) learning based methods (Chrupala et al, 2008;Toutanova and Cherry, 2009;Gesmundo and Samardzic, 2012;Müller et al, 2015;Nicolai and Kondrak, 2016) requiring prior training dataset to learn the morphological patterns. Again, the later methods can be further classified depending on whether context of the current word is considered or not.…”

Low-Resource G2P and P2G Conversion with Synthetic Training Data

“…In DirecTL+ (Jiampojamarn et al, 2010), the feature set was augmented with joint n-grams defined on both source and target substrings. The system was applied to related tasks such as transliteration (Jiampojamarn et al, 2009), morphological inflection (Nicolai et al, 2015), stemming (Nicolai and Kondrak, 2016), and cognate projection (Hauer et al, 2019), proving to be particularly competitive in low-resource settings. DTLM (Nicolai et al, 2018), our principal tool in this work, is a successor of DirecTL+, which incorporates target-side language models and a highprecision alignment.…”

A Simple Joint Model for Improved Contextual Neural Lemmatization

“…Previous work on lemmatization has investigated both neural (Bergmanis and Goldwater, 2019) and non-neural (Chrupała, 2008;Müller et al, 2015;Nicolai and Kondrak, 2016;Cotterell et al, 2017) methods. We compare our approach against recent competing methods that report results on UD datasets.…”