Indexing and stemming approaches for the Czech language

Abstract

a b s t r a c tThis paper describes and evaluates various stemming and indexing strategies for the Czech language. Based on Czech test-collection, we have designed and evaluated two stemming approaches, a light and a more aggressive one. We have compared them with a no stemming scheme as well as a language-independent approach (n-gram). To evaluate the suggested solutions we used various IR models, including Okapi, Divergence from Randomness (DFR), a statistical language model (LM) as well as the classical tf idf vector-space approach. We found that the Divergence from Randomness paradigm tend to propose better retrieval effectiveness than the Okapi, LM or tf idf models, the performance differences were however statistically significant only with the last two IR approaches. Ignoring the stemming reduces generally the MAP by more than 40%, and these differences are always significant. Finally, if our more aggressive stemmer tends to show the best performance, the differences in performance with a light stemmer are not statistically significant.

Comparative Study of Indexing and Search Strategies for the Hindi, Marathi, and Bengali Languages

“…Based on this data, we found that a more aggressive stemmer tended to result in better MAP while for some languages (e.g., Bengali) the performance difference between a light and an aggressive stemmer was not significant. Moreover, when compared to MAP found for certain European languages, these relative improvements after stemming for these three Indian languages were quite large (e.g., 4% for English, 4.1% for Dutch, 7% for Spanish, 9% for French, 15% for Italian, 19% for German, 29% for Swedish, 40% for Finnish [Tomlinson 2004], and 45% for Czech [Dolamic and Savoy 2010]). …”

“…A noun's inflectional termination depends on its case, number, and gender, thus resulting in the complex morpho-syntaxical construction often found in other Indo-European languages, such as Czech [Dolamic and Savoy 2010].…”

“…For the three Indian languages we created light stemmers, the same strategy we have suggested for other European languages over the past few years [Savoy 2006;Dolamic and Savoy 2010]. In our opinion effective stemming should focus mainly on inflectional suffixes attached to nouns and adjectives (used to sustain most of a document's meaning) and ignore numerous verb forms.…”

Reprint of “Supervised sentiment analysis in Czech social media”

“…14 In many NLP applications, a very popular preprocessing technique is stemming. We tested the Czech light stemmer (Dolamic & Savoy, 2009) and High Precision Stemmer. 15 Another widely-used method for reducing the vocabulary size, and thus the feature space, is lemmatization.…”

Ad Hoc Retrieval with the Persian Language

“…For some languages (e.g., Chinese, Japanese [1]), word segmentation is not an easy task, while for others (e.g., German), the use of different compound constructions to express the same concept or idea may hurt the retrieval quality [2]. The presence of numerous inflectional suffixes (e.g., Hungarian [3], Finnish), even for names (e.g., Czech [4], Russian [5]) as well as numerous derivational suffixes must be taken into account for an effective retrieval.…”