Roman to Gurmukhi Social Media Text Normalization

Abstract

PurposeNormalization is an important step in all the natural language processing applications that are handling social media text. The text from social media poses a different kind of problems that are not present in regular text. Recently, a considerable amount of work has been done in this direction, but mostly in the English language. People who do not speak English code mixed the text with their native language and posted text on social media using the Roman script. This kind of text further aggravates the problem of normalizing. This paper aims to discuss the concept of normalization with respect to code-mixed social media text, and a model has been proposed to normalize such text.Design/methodology/approachThe system is divided into two phases – candidate generation and most probable sentence selection. Candidate generation task is treated as machine translation task where the Roman text is treated as source language and Gurmukhi text is treated as the target language. Character-based translation system has been proposed to generate candidate tokens. Once candidates are generated, the second phase uses the beam search method for selecting the most probable sentence based on hidden Markov model.FindingsCharacter error rate (CER) and bilingual evaluation understudy (BLEU) score are reported. The proposed system has been compared with Akhar software and RB\_R2G system, which are also capable of transliterating Roman text to Gurmukhi. The performance of the system outperforms Akhar software. The CER and BLEU scores are 0.268121 and 0.6807939, respectively, for ill-formed text.Research limitations/implicationsIt was observed that the system produces dialectical variations of a word or the word with minor errors like diacritic missing. Spell checker can improve the output of the system by correcting these minor errors. Extensive experimentation is needed for optimizing language identifier, which will further help in improving the output. The language model also seeks further exploration. Inclusion of wider context, particularly from social media text, is an important area that deserves further investigation.Practical implicationsThe practical implications of this study are: (1) development of parallel dataset containing Roman and Gurmukhi text; (2) development of dataset annotated with language tag; (3) development of the normalizing system, which is first of its kind and proposes translation based solution for normalizing noisy social media text from Roman to Gurmukhi. It can be extended for any pair of scripts. (4) The proposed system can be used for better analysis of social media text. Theoretically, our study helps in better understanding of text normalization in social media context and opens the doors for further research in multilingual social media text normalization.Originality/valueExisting research work focus on normalizing monolingual text. This study contributes towards the development of a normalization system for multilingual text.

Construction and Application of Three-dimensional Information Management System for Intelligent Buildings Integrating BIM and GIS Technologies

“…The gravity center of the model is shifted to the origin of the coordinate system to achieve normalization. The x-axis coordinates of the centre of gravity of the model are calculated as shown in equation 3.1 [14,16].…”

Contextualized dynamic meta embeddings based on Gated CNNs and self-attention for Arabic machine translation

“…Each of these layers consists of: a multi-head self-attention mechanism sub-layer, which is followed by a position-wise fully connected feed-forward network sub-layer. Furthermore, for each of the two sublayers, a residual connection (He et al, 2016) is employed, followed by a normalization layer (Ba et al, 2016). We have modified the architecture of the Transformer model to enhance its performance and address specific challenges in NMT.…”

Idea plagiarism detection with recurrent neural networks and vector space model

“…Yet, the nominal amount of work has been done in the domain of intelligent plagiarism so far. The recent advances in the field of machine learning (Fei et al, 2019) and deep learning have paved the way for new techniques (Verhelst and Moons, 2017;Kaur and Singh, 2020) and methodologies to detect intelligent plagiarism, including paraphrasing and summarization. This paper aims to introduce one such state of the art technique.…”