Morphological Analyzer for Malayalam Using Machine Learning

Abeera, V. P.; Aparna, S.; Rekha, R.; Kumar, M. Anand; Dhanalakshmi, V.; Rajendran, Suresh

doi:10.1007/978-3-642-27872-3_38

Cited by 7 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But for the reasons listed below, none of these could be used for the language modeling task required for ASR. For Malayalam morphological tokenization, earlier studies have used probabilistic, rule-based suffix-stripping, machine learning, and dictionary-based approaches [28][29][30][31]. The most recent deep learning technique uses Romanised Malayalam text and requires annotated data for training [32].…”

Section: Subword Tokenization Algorithmsmentioning

confidence: 99%

Improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling

Manohar,

A R,

Rajan

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

This article presents the research work on improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling. The speech recognition system is built using a deep neural network–hidden Markov model (DNN-HMM)-based automatic speech recognition (ASR). We propose a novel method, syllable-byte pair encoding (S-BPE), that combines linguistically informed syllable tokenization with the data-driven tokenization method of byte pair encoding (BPE). The proposed method ensures words are always segmented at valid pronunciation boundaries. On a text corpus that has been divided into tokens using the proposed method, we construct statistical n-gram language models and assess the modeling effectiveness in terms of both information-theoretic and corpus linguistic metrics. A comparative study of the proposed method with other data-driven (BPE, Morfessor, and Unigram), linguistic (Syllable), and baseline (Word) tokenization algorithms is also presented. Pronunciation lexicons of subword tokenized units are built with pronunciation described as graphemes. We develop ASR systems employing the subword tokenized language models and pronunciation lexicons. The resulting ASR models are comprehensively evaluated to answer the research questions regarding the impact of subword tokenization algorithms on language modeling complexity and on ASR performance. Our study highlights the strong performance of the hybrid S-BPE tokens, achieving a notable 10.6% word error rate (WER), which represents a substantial 16.8% improvement over the baseline word-level ASR system. The ablation study has revealed that the performance of S-BPE segmentation, which initially underperformed compared to syllable tokens with lower amounts of textual data for language modeling, exhibited steady improvement with the increase in LM training data. The extensive ablation study indicates that there is a limited advantage in raising the n-gram order of the language model beyond $$n=3$$ n = 3 . Such an increase results in considerable model size growth without significant improvements in WER. The implementation of the algorithm and all associated experiments are available under an open license, allowing for reproduction, adaptation, and reuse.

show abstract

Section: Subword Tokenization Algorithmsmentioning

confidence: 99%

Improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling

Manohar,

A R,

Rajan

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

show abstract

“…In paper by V P Abeera,Et.al [10],is a two step process. Initially it is morphological data creation for Malayalam language and the next stage is implementation of morphological analyzer.…”

Section: Related Workmentioning

confidence: 99%

“…In a system developed by Anand kumar M, Et.al [16] make use of machine learning in Tamil language which is similar to [10] . [7] ii.…”

Section: Related Workmentioning

confidence: 99%

Morphological Analyzer for Malayalam: A Literature Survey

Shaji¹,

Sindhu²

2014

IJCA

View full text Add to dashboard Cite

Natural Language processing deals with analysing, generating and understanding the languages that human generally use. Morphological Analyzer is an important part in Natural Language Processing. Morphological Analyzer always returns the morpheme and its associated grammatical structure. This paper describes about the different techniques in morphological analyzer and different implementations of morphological analyzer in Malayalam.

show abstract