How low is too low? A monolingual take on lemmatisation in Indian languages

Abstract

Lemmatization aims to reduce the sparse data problem by relating the inflected forms of a word to its dictionary form. Most prior work on ML based lemmatization has focused on high resource languages, where data sets (word forms) are readily available. For languages which have no linguistic work available, especially on morphology or in languages where the computational realization of linguistic rules is complex and cumbersome, machine learning based lemmatizers are the way to go. In this paper, we devote our attention to lemmatisation for low resource, morphologically rich scheduled Indian languages using neural methods. Here, low resource means only a small number of word forms are available. We perform tests to analyse the variance in monolingual models' performance on varying the corpus size and contextual morphological tag data for training. We show that monolingual approaches with data augmentation can give competitive accuracy even in the low resource setting, which augurs well for NLP in low resource setting.

BanLemma: A Word Formation Dependent Rule and Dictionary Based Bangla Lemmatizer

“…Release of the Universal Dependencies (UD) dataset (de Marneffe et al, 2014,Nivre et al, 2017 and Sigmorphon 2019 shared task formed the basis of encoder-decoder architectures to solve the lemmatization task as a string-transduction task (Qi et al, 2018; Kanerva et al, 2018. For the Bangla language, Saunack et al (2021) employed a similar two-step attention network that took morphological tags and inflected words as input. Islam et al (2022) used PoS tags of each word as additional features to the encoder-decoder network achieving 95.75% accuracy on validation dataset.…”

Lemmatization Experiments on Two Low-Resourced Languages: Low Saxon and Occitan

“…Bergmanis and Goldwater (2018) evaluate their models both on the full amount of available data and on 10k samples. Saunack et al (2021) explore the lower bound for training data size on Indian languages: they compare a standard setting with low-resource settings with only 500 and 100 training instances, in which they rely on data augmentation techniques. Saurav et al (2020) investigate cross-lingual approaches for lemmatizing low-resourced Indian languages.…”

Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)

“…Bergmanis and Goldwater (2018) evaluate their models both on the full amount of available data and on 10k samples. Saunack et al (2021) explore the lower bound for training data size on Indian languages: they compare a standard setting with low-resource settings with only 500 and 100 training instances, in which they rely on data augmentation techniques. Saurav et al (2020) investigate cross-lingual approaches for lemmatizing low-resourced Indian languages.…”