Improving stemming for Arabic information retrieval

Larkey, Leah S.; Ballesteros, Lisa; Connell, Margaret E.

doi:10.1145/564376.564425

Cited by 251 publications

(155 citation statements)

References 19 publications

Supporting

Mentioning

139

Contrasting

Order By: Relevance

“…This indicates that when queries are processed using language modeling techniques, they yield documents that are more relevant compared to queries which are not processed. This is similar with studies that have reported language models to improve document retrievals [6]- [11].…”

Section: Evaluation Metricssupporting

confidence: 89%

“…The process is used in removing derivational suffixes as well as inflections (i.e. suffixes that change the form of words and their grammatical functions) so that word variants can be conflated into the same roots or stems.Stemming mechanisms have been used in a lot of language research areas such as Arabic [6] II. RELATED WORK cross-lingual retrieval [7] and multi-language manipulations [8].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stemming and Lemmatization: A Comparison of Retrieval Performances

Balakrishnan¹,

Lloyd-Yemoh²

2014

LNSE

154

View full text Add to dashboard Cite

Abstract-The current study proposes to compare document retrieval precision performances based on language modeling techniques, particularly stemming and lemmatization. Stemming is a procedure to reduce all words with the same stem to a common form whereas lemmatization removes inflectional endings and returns the base or dictionary form of a word. Comparisons were also made between these two techniques with a baseline ranking algorithm (i.e. with no language processing). A search engine was developed and the algorithms were tested based on a test collection. Both mean average precisions and histograms indicate stemming and lemmatization to outperform the baseline algorithm. As for the language modeling techniques, lemmatization produced better precision compared to stemming, however the differences are insignificant. Overall the findings suggest that language modeling techniques improves document retrieval, with lemmatization technique producing the best result.

show abstract

Section: Evaluation Metricssupporting

confidence: 89%

Section: Introductionmentioning

confidence: 99%

Stemming and Lemmatization: A Comparison of Retrieval Performances

Balakrishnan¹,

Lloyd-Yemoh²

2014

LNSE

154

View full text Add to dashboard Cite

show abstract

“…Refer to (Buckwalter, 2005) and (Larkey et al, 2002) for examples that highlight morphological inflection for a simple Modern Standard Arabic (MSA) word and basic stemming operations that we use as our baseline system. (Nießen and Ney, 2000) tackle the inflection gap for German-to-English word alignment by performing a series of morphological operations on the German text.…”

Section: Introductionmentioning

confidence: 99%

Bridging the inflection morphology gap for Arabic statistical machine translation

Zollmann

Venugopal

Vogel

2006

Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers on XX - NAACL '06

View full text Add to dashboard Cite

Statistical machine translation (SMT) is based on the ability to effectively learn word and phrase relationships from parallel corpora, a process which is considerably more difficult when the extent of morphological expression differs significant across the source and target languages. We present techniques that select appropriate word segmentations in the morphologically rich source language based on contextual relationships in the target language. Our results take advantage of existing word level morphological analysis components to improve translation quality above state-of-the-art on a limited-data Arabic to English speech translation task.

show abstract

“…It proceeds first by a normalization that converts the document into a manipulable format [15]. This step is a delicate task because Arabic is inflectional and strongly derivational language [5].…”

Section: Our Methodsmentioning

confidence: 99%