Comparative Study of Indexing and Search Strategies for the Hindi, Marathi, and Bengali Languages

Abstract

The main goal of this article is to describe and evaluate various indexing and search strategies for the Hindi, Bengali, and Marathi languages. These three languages are ranked among the world's 20 most spoken languages and they share similar syntax, morphology, and writing systems. In this article we examine these languages from an Information Retrieval (IR) perspective through describing the key elements of their inflectional and derivational morphologies, and suggest a light and more aggressive stemming approach based on them.In our evaluation of these stemming strategies we make use of the FIRE 2008 test collections, and then to broaden our comparisons we implement and evaluate two language independent indexing methods: the n-gram and trunc-n (truncation of the first n letters). We evaluate these solutions by applying our various IR models, including the Okapi, Divergence from Randomness (DFR) and statistical language models (LM) together with two classical vector-space approaches: tf idf and Lnu-ltc.Experiments performed with all three languages demonstrate that the I(n e )C2 model derived from the Divergence from Randomness paradigm tends to provide the best mean average precision (MAP). Our own tests suggest that improved retrieval effectiveness would be obtained by applying more aggressive stemmers, especially those accounting for certain derivational suffixes, compared to those involving a light stemmer or ignoring this type of word normalization procedure. Comparisons between no stemming and stemming indexing schemes shows that performance differences are almost always statistically significant. When, for example, an aggressive stemmer is applied, the relative improvements obtained are ∼28% for the Hindi language, ∼42% for Marathi, and ∼18% for Bengali, as compared to a no-stemming approach. Based on a comparison of word-based and language-independent approaches we find that the trunc-4 indexing scheme tends to result in performance levels statistically similar to those of an aggressive stemmer, yet better than the 4-gram indexing scheme. A query-by-query analysis reveals the reasons for this, and also demonstrates the advantage of applying a stemming or a trunc-4 indexing scheme.

A novel corpus-based stemming algorithm using co-occurrence statistics

“…A number of works have reported the effectiveness of light stemming approaches [20,4] for a variety of languages both European and Asian. As opposed to the stemmers like Porter, light stemming methods focus on removing only inflectional variants from the end of a word.…”

“…al. [4] is chosen as the rule based approach for Marathi and Bengali. The Hungarian stemmer is by Savoy [21] and the Czech stemmer is by Dolamic et.…”

Automatic Classification of Bengali Sentences Based on Sense Definitions Present in Bengali Wordnet

“…The adjectives, on the other hand, are normally inflected with some primary and secondary adjectival suffixes denoting degree, quality, quantity, and similar other attributes. As a result, to build up a complete and robust system for WSD for all types of morphologically derived forms tagged with lexical information and semantic relations is a real challenge for a language like Bengali [15][16][17][18][19][20][21][22][23][24][25].…”

“…In our approach, we have therefore used three types of learning sets that are built up according to three different meanings of the ambiguous word (māthā) "head", which are collected from the Bengali WordNet. There are five types of dictionary definition (glosses) for the word with twenty five (25) ), 2 nd and 3 rd category represent second type of meaning ( = ), 4 th and 5 th category represent third type of meaning ( = = । ). Based on the information types we have built up three specific categories of senses of (māthā): (a) " , ", (b) " , " and (c) " , , । ".…”

Exploring the Partial Textual Entailment Problem for Bengali News Texts

“…Some of the most frequent Bengali suffixes are 'ই', 'গুবলা', 'র্া', 'টি', 'রা', 'হীন'. We have used the system described in the [4], with some simple additional modifications.…”