Dealing with word-internal modification and spelling variation in data-driven lemmatization

Abstract

This paper describes our contribution to two challenges in data-driven lemmatization. We approach lemmatization in the framework of a two-stage process, where first lemma candidates are generated and afterwards a ranker chooses the most probable lemma from these candidates. The first challenge is that languages with rich morphology like Modern German can feature morphological changes of different kinds, in particular word-internal modification. This makes the generation of the correct lemma a harder task than just removing suffixes (stemming). The second challenge that we address is spelling variation as it appears in non-standard texts. We experiment with different generators that are specifically tailored to deal with these two challenges. We show in an oracle setting that there is a possible increase in lemmatization accuracy of 14% with our methods to generate lemma candidates on Middle Low German, a group of historical dialects of German (1200-1650 AD). Using a log-linear model to choose the correct lemma from the set, we obtain an actual increase of 5.56%.

Detecting spelling variants in non-standard texts

“…This approach has been followed by Kestemont et al (2010) and Barteld et al (2016) for lemmatization of historical texts.…”

“…The candidate generation used in the original approach cannot generate the correct candidate for spelling variants of non-standard words that did not appear in the training data, e.g., you are as a normalization candidate for urr will not be generated if only ur as a non-standard variant of you are is known from the training data. Similarly to the approach that Barteld et al (2016) used to improve the lemmatization of non-standard texts, the knowledge that urr is a spelling variant of ur could be used to generate the candidate you are and thereby improve the coverage of the generator.…”

“…Alternatively, specialized tools can be developed that directly use the knowledge about spelling variation. This approach has been followed by Kestemont et al (2010) and Barteld et al (2016) for lemmatization of historical texts.…”

Investigating Diatopic Variation in a Historical Corpus

“…Baron et al (2009)). However, the derived mappings themselves that map historical to modern word forms are usually not in the focus of interest (but see Barteld et al (2016)). …”