TrieIR: Indexing and Retrieval Engine for Kannada Unicode Text

Kulkarni, Sumant; Srinivasa, Srinath

doi:10.1007/978-3-319-03599-4_3

“…Our experiments are conducted with python implementation of V-KTPY encryption and VKTPY Family of Tries. The data set used for experiment are set of short political articles penned by noted journalist Shekhar Gupta for his 'Prajavani'(A famous Kannada daily) column 'RaaShtrakaaraNa' [20]. These articles contains 6200 words after removal of stops words by our own tool [22].…”

Section: Experimental Scenarios and Resultsmentioning

confidence: 99%

Novel Space Efficient Indices for Kannada Text: VKTPY Trie Family

2019

IJEAT

1

0

View full text Add to dashboard Cite

V-KTPY Trie family is a group of space efficient indices designed to store and search Kannada text. The existing text searching and document fetching methods use different kinds of indices such as indices based on hashing, lexicographical indices and clustering technique based indexing. Each of these indexing methods have their own advantages including the optimal time complexity. However these indices are not space efficient. In this paper we are proposing a family of novel space efficient indices called V-KTPY Tries, which have the features of both lexicographical and hash based indexing. V-KTPY Tries are congruence of V-KTPY Rule (“Vistruta Katapayadi sutra”) and Prefix trees (Trie) , where the text labels of the Trie are encrypted by V-KTPY Rule. This powerful rule is an extension of an ancient “Katapayadi Sutra” (KTPY Rule) which can convert characters of Brahmi/Devanagari scripts to numbers. In this paper V-KTPY Tries are indexing V-KTPY encrypted Kannada text due to which compression is possible. The experiments are conducted on the family of V-KTPY Tries and their corresponding Tries with unicode Kannada. And the results show that the simple V-KTPY Trie gives 35% space efficiency; V-KTPY 10Ary Trie gives 65% space efficiency over simple unicode Trie with almost the same time complexity. The Prefix Hashed Trie is a fully compressed V-KTPY Trie which gives 20% space efficiency when compared to fully compressed unicode Trie. V-KTPY Tries can be used where Tries are applicable. The VKTPY prefix hashed Tries are used in Kannada feature selection. V-KTPY Tries can be extended to index many (120+) Indian languages which follow Brahmi or Devanagari script.

show abstract

“…For example, the word ೕಕು (beku -want) is agglutinated with the word ಆಗು (Agu -become) as a suffix, it forms the word ೕ ಾಗು (bekAgu -wanted). Inflectional suffixes are morphemes which are added to root words to produce grammatical variants of [15]. In addition to this, Kannada grammar also supports morphophonemic processes (called Sandhi) resulting in changes of sounds when two or more words and morphemes join to form a new word.…”

Section: Challenges Of Named Entity Recognition In Kannada Languagementioning

confidence: 99%

Active learning for Named Entity Recognition in Kannada

M,

Srinivasa

2023

Preprint

0

View full text Add to dashboard Cite

<p>Named Entity Recognition (NER) task aims at automatically recognising and classifying named entities in a given natural language input. Majority of the studies related to Named Entity Recognition are focused on English language. Named Entity Recognition in Indian languages is challenging due to the complex grammar and morphology of Indian languages and the scarcity of good quality labelled data. The dissimilarity between the literary and spoken versions of these languages is also a big challenge regarding usability of NER models. Kannada is such an Indian language for which the task of NER is still an active area of research. Usage of Deep learning, especially Transfer learning using Transformer models has drastically improved performance of many NLP tasks at scale. However, Transfer learning still requires considerable amount of data for the required task. In case of low resource language like Kannada, very few labelled datasets are available publicly and creating one from scratch is expensive in terms of time and labor. Active learning (AL) aims to tackle the labelled data acquisition problem by having the learning model and an Oracle to cooperate. Active Learning iteratively builds an optimally labelled and sufficiently large dataset. This study focuses on Named Entity Recognition in Kannada language. We explore the application of Active Learning technique to Named Entity Recognition problem in the low-resource language Kannada. Results show that AL can be used to boost a multilingual models performance in fine-tuning for NER. We also try to mitigate the gap between formal and colloquial dialects of Kannada in NER datasets.</p>

show abstract

“…For example, the word ೕಕು (beku -want) is agglutinated with the word ಆಗು (Agu -become) as a suffix, it forms the word ೕ ಾಗು (bekAgu -wanted). Inflectional suffixes are morphemes which are added to root words to produce grammatical variants of [15]. In addition to this, Kannada grammar also supports morphophonemic processes (called Sandhi) resulting in changes of sounds when two or more words and morphemes join to form a new word.…”

Section: Challenges Of Named Entity Recognition In Kannada Languagementioning

confidence: 99%

Active learning for Named Entity Recognition in Kannada

M,

Srinivasa

2023

Preprint

0

View full text Add to dashboard Cite

<p>Named Entity Recognition (NER) task aims at automatically recognising and classifying named entities in a given natural language input. Majority of the studies related to Named Entity Recognition are focused on English language. Named Entity Recognition in Indian languages is challenging due to the complex grammar and morphology of Indian languages and the scarcity of good quality labelled data. The dissimilarity between the literary and spoken versions of these languages is also a big challenge regarding usability of NER models. Kannada is such an Indian language for which the task of NER is still an active area of research. Usage of Deep learning, especially Transfer learning using Transformer models has drastically improved performance of many NLP tasks at scale. However, Transfer learning still requires considerable amount of data for the required task. In case of low resource language like Kannada, very few labelled datasets are available publicly and creating one from scratch is expensive in terms of time and labor. Active learning (AL) aims to tackle the labelled data acquisition problem by having the learning model and an Oracle to cooperate. Active Learning iteratively builds an optimally labelled and sufficiently large dataset. This study focuses on Named Entity Recognition in Kannada language. We explore the application of Active Learning technique to Named Entity Recognition problem in the low-resource language Kannada. Results show that AL can be used to boost a multilingual models performance in fine-tuning for NER. We also try to mitigate the gap between formal and colloquial dialects of Kannada in NER datasets.</p>

show abstract

TrieIR: Indexing and Retrieval Engine for Kannada Unicode Text

Cited by 3 publications

References 2 publications

Novel Space Efficient Indices for Kannada Text: VKTPY Trie Family

Novel Space Efficient Indices for Kannada Text: VKTPY Trie Family

Active learning for Named Entity Recognition in Kannada

Active learning for Named Entity Recognition in Kannada

Contact Info

Product

Resources

About