Design of rule based lemmatizer for Kannada inflectional words

Natural language is a crucial tool to facilitate communication in our day-to-day activities. This can be achieved either in text or speech forms. Natural language processing (NLP) involves making computers understand and process natural language. NLP has enhanced the way humans interact with computers, from having computers use speech to talk to humans as well as having computers translate human speech. Apart from speech, computers also create and understand sentences in natural language in a process called morphological analysis. Morphological analysis is an important part in natural language processing, being applied as a preprocessing step in most NLP tasks. Morphological analysis consists of four subtasks, that is, lemmatization, part-of-speech (POS) tagging, word segmentation and stemming. In this paper, we explore in detail each of these tasks of morphological analysis. We then evaluate the techniques used in this NLP field. Finally, we give a summary of the results of each of these techniques.

show abstract

“…A rule-based lemmatizer or Kannada is proposed by (Prathibha & Padma, 2016). A manual dictionary of both verb and noun roots was created.…”

Section: Machine Learningmentioning

confidence: 99%

review of techniques for morphological analysis in natural language processing

Muthee

Makau²,

Amos³

2022

AJSTSS

View full text Add to dashboard Cite

show abstract

“…When the complete set of morphological features is included in training, most languages achieve extremely high accuracy (at least 95%, except for Kannada), even when data set sizes are as small as 1000. When the data set size is 500, the accuracy drop to the range 80-90% but are still competitive wrt rulebased lemmatisers across languages (Bhattacharyya et al, 2014) like Sanskrit(Raulji and Saini, 2019), Hindi (Paul et al, 2013), Bengali(Shakib et al, 2019), Urdu (Gupta et al, 2015) and Kannada (Prathibha and Padma, 2015). However, the performance drops drastically when the data set size is reduced to 100.…”

Section: Variation With Number Of Training Word-pairsmentioning

confidence: 99%

How low is too low? A monolingual take on lemmatisation in Indian languages

Kumar¹,

Kumar²,

Bhattacharyya³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Lemmatization aims to reduce the sparse data problem by relating the inflected forms of a word to its dictionary form. Most prior work on ML based lemmatization has focused on high resource languages, where data sets (word forms) are readily available. For languages which have no linguistic work available, especially on morphology or in languages where the computational realization of linguistic rules is complex and cumbersome, machine learning based lemmatizers are the way to go. In this paper, we devote our attention to lemmatisation for low resource, morphologically rich scheduled Indian languages using neural methods. Here, low resource means only a small number of word forms are available. We perform tests to analyse the variance in monolingual models' performance on varying the corpus size and contextual morphological tag data for training. We show that monolingual approaches with data augmentation can give competitive accuracy even in the low resource setting, which augurs well for NLP in low resource setting.

show abstract

“…Lemmatization extracts the original form of a word as the word is converted to a basic form. erefore, lemmatization does not change the meaning of words [23][24][25].…”

Section: Text Data Preprocessingmentioning

confidence: 99%

Validation of Text Data Preprocessing Using a Neural Network Model

Woo

Kim

Lee

2020

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Many artificial intelligence studies focus on designing new neural network models or optimizing hyperparameters to improve model accuracy. To develop a reliable model, appropriate data are required, and data preprocessing is an essential part of acquiring the data. Although various studies regard data preprocessing as part of the data exploration process, those studies lack awareness about the need for separate technologies and solutions for preprocessing. Therefore, this study evaluated combinations of preprocessing types in a text-processing neural network model. Better performance was observed when two preprocessing types were used than when three or more preprocessing types were used for data purification. More specifically, using lemmatization and punctuation splitting together, lemmatization and lowering together, and lowering and punctuation splitting together showed positive effects on accuracy. This study is significant because the results allow better decisions to be made about the selection of the preprocessing types in various research fields, including neural network research.

show abstract

Design of rule based lemmatizer for Kannada inflectional words

Cited by 16 publications

References 6 publications

review of techniques for morphological analysis in natural language processing

review of techniques for morphological analysis in natural language processing

How low is too low? A monolingual take on lemmatisation in Indian languages

Validation of Text Data Preprocessing Using a Neural Network Model

Contact Info

Product

Resources

About