Unsupervised learning of morphology for building lexicon for a highly inflectional language

A Framework for Learning Morphology using Suffix Association Matrix

“…Here the restriction on stem-length first proposed by Gaussier is upheld. Sharma's (2006) work deals with neutral suffix only and does not capture nonneutral suffixes. These studies are limited to suffix identification and do not generate paradigms.…”

“…One promising methodology for unsupervised segmentation which does not make any suffix frequency assumptions is p-similar technique for morpheme segmentation first proposed by Gaussier (1999). Researchers have used this method for suffix identification and not for segmentation (Gaussier, 1999;Sharma, 2006). We extended this less studied technique to segment words by introducing the concept of suffix association matrix, thus giving us an unsupervised method which correctly identifies suffixes irrespective of their frequency of occurrence in the corpus and also segments short stem words.…”

“…We extended this less studied technique to segment words by introducing the concept of suffix association matrix, thus giving us an unsupervised method which correctly identifies suffixes irrespective of their frequency of occurrence in the corpus and also segments short stem words. To the best of our knowledge, most reported work which uses p-similar technique for suffix identification (Gaussier, 1999;Sharma, 2006) enforce a restriction on stem-length that it should be at least five. This restriction works well for suffix identification but not for segmentation.…”

A naive theory of affixation and an algorithm for extraction

“…Many publications (Ćavar et al, 2004;Brent et al, 1995;Déjean, 1998;Argamon et al, 2004;Goldsmith, 2001;Creutz and Lagus, 2005;Neuvel and Fulop, 2002;Baroni, 2003;Gaussier, 1999;Sharma et al, 2002;Wicentowski, 2002;Oliver, 2004), and various other works by the same authors, describe strategies that use frequencies, probabilities, and optimization criteria, often Minimum Description Length (MDL), in various combinations. So far, all these are unsatisfactory on two main accounts; on the theretical side, they still owe an explanation of why compression or MDL should give birth to segmentations coinciding with morphemes as linguistically defined.…”

“…Secondly, segmentation algorithms may have different purposes and it might not make good sense to study segmentation in isolation from induction of paradigms. Lastly, and most importantly, all of the reviewed techniques (Wicentowski, 2004;Wicentowski, 2002;Baroni et al, 2002;Andreev, 1965;Ćavar et al, 2004;Snover and Brent, 2003;Snover and Brent, 2001;Schone and Jurafsky, 2001;Jacquemin, 1997;Goldsmith and Hu, 2004;Sharma et al, 2002;Clark, 2001;Kazakov and Manandhar, 1998;Déjean, 1998;Oliver, 2004;Creutz and Lagus, 2003;Creutz and Lagus, 2004;Hirsimäki et al, 2003;Creutz and Lagus, 2005;Argamon et al, 2004;Gaussier, 1999;Lehmann, 1973;Langer, 1991;Flenner, 1995;Klenk and Langer, 1989;Goldsmith, 2001;Goldsmith, 2000;Hu et al, 2005b;Hu et al, 2005a;Brent et al, 1995), as they are described, have threshold-parameters of some sort, explicitly claim not to work well for an open set of languages, or require noise-free all-form input (Albright, 2002;Manning, 1998;Borin, 1991). Therefore it is not possible to even design a fair test.…”