SS4MCT: A Statistical Stemmer for Morphologically Complex Texts

Abstract

There have been multiple attempts to resolve various inflection matching problems in information retrieval. Stemming is a common approach to this end. Among many techniques for stemming, statistical stemming has been shown to be effective in a number of languages, particularly highly inflected languages. In this paper we propose a method for finding affixes in different positions of a word. Common statistical techniques heavily rely on string similarity in terms of prefix and suffix matching. Since infixes are common in irregular/informal inflections in morphologically complex texts, it is required to find infixes for stemming. In this paper we propose a method whose aim is to find statistical inflectional rules based on minimum edit distance table of word pairs and the likelihoods of the rules in a language. These rules are used to statistically stem words and can be used in different text mining tasks. Experimental results on CLEF 2008 and CLEF 2009 English-Persian CLIR tasks indicate that the proposed method significantly outperforms all the baselines in terms of MAP.

An Analytical Analysis of Text Stemming Methodologies in Information Retrieval and Natural Language Processing Systems

“…Pande et al [62] also used an N-gram technique to develop a stemmer and frequency of the N-gram to determine the stem's possibility. Dadashkarimi et al [63] proposed a statistical stemmer to extract the root from the inflectional and derivational forms of the word. The method learns the linguistic pattern from the huge collection of web pages using Minimum Edit Distance (MED) algorithms and Parts of Speech (POS) tagger.…”

“…Ferilli et al [65] tested the stemmer on English, Italian, French, and Latin text. Statistical stemmers, evaluated in the context of IR systems such as Dadashkarimi et al [63] stemmer are shown in Table 15. Statistical stemmer's performance may vary from language to language as mentioned in Table 15, for example, a stemmer has achieved 92.63% accuracy for the Italian language whereas it achieved 89.54 for the French language.…”