Language Detection Engine for Multilingual Texting on Mobile Devices

Gothe, Sourabh Vasant; Ghosh, Sourav; Mani, Sharmila; Bhanodai, Guggilla; Agarwal, Ankur; Sanchi, Chandramouli

doi:10.1109/icsc.2020.00057

Cited by 3 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These further make it difficult for an on-device language model to accurately map a user typed token to the intended native vocabulary word. Such issues also impede the end-user experience with Multilingual Texting and Language Detection [6]. In the present work, we address these problems by proposing a novel disambiguation algorithm for two mutually nonexclusive input methods [7] for languages with abugida scripts.…”

Section: Introductionmentioning

confidence: 99%

edATLAS: An Efficient Disambiguation Algorithm for Texting in Languages with Abugida Scripts

Ghosh

Gothe

Sanchi

et al. 2021

2021 IEEE 15th International Conference on Semantic Computing (ICSC)

Self Cite

View full text Add to dashboard Cite

Abugida refers to a phonogram writing system where each syllable is represented using a single consonant or typographic ligature, along with a default vowel or optional diacritic(s) to denote other vowels. However, texting in these languages has some unique challenges in spite of the advent of devices with soft keyboard supporting custom key layouts. The number of characters in these languages is large enough to require characters to be spread over multiple views in the layout. Having to switch between views many times to type a single word hinders the natural thought process. This prevents popular usage of native keyboard layouts. On the other hand, supporting romanized scripts (native words transcribed using Latin characters) with language model based suggestions is also set back by the lack of uniform romanization rules.To this end, we propose a disambiguation algorithm and showcase its usefulness in two novel mutually non-exclusive input methods for languages natively using the abugida writing system: (a) disambiguation of ambiguous input for abugida scripts, and (b) disambiguation of word variants in romanized scripts. We benchmark these approaches using public datasets, and show an improvement in typing speed by 19.49%, 25.13%, and 14.89%, in Hindi, Bengali, and Thai, respectively, using Ambiguous Input, owing to the human ease of locating keys combined with the efficiency of our inference method. Our Word Variant Disambiguation (WDA) maps valid variants of romanized words, previously treated as Out-of-Vocab, to a vocabulary of 100k words with high accuracy, leading to an increase in Error Correction F1 score by 10.03% and Next Word Prediction (NWP) by 62.50% on average.

show abstract

Section: Introductionmentioning

confidence: 99%