Language identification framework in code-mixed social media text based on quantum LSTM — the word belongs to which language?

Shekhar, Shashi; Sharma, Dilip Kumar; Beg, M. M. Sufyan

doi:10.1142/s0217984920500864

Cited by 32 publications

(14 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent studies [35]- [37] reveal the utility of codeswitching in Hindi-English language pairs, from Twitter, Facebook, and WhatsApp social media sites. Vivek et al [38] proposed the generation of a candidate sentence as feature generation, for the Bidirectional Long Short Term Memory (Bi-LSTM) -based neural network, to classify Hindi-English code-mixed texts into three labels: positive, negative and neutral.…”

Section: Related Workmentioning

confidence: 99%

Transformer Based Language Identification for Malayalam-English Code-Mixed Text

Thara

Poornachandran

2021

IEEE Access

View full text Add to dashboard Cite

Social media users have the proclivity to write majority of the data for under resourced languages in code-mixed format. Code-mixing is defined as mixing of two or more languages in a single sentence. Research in code-mixed text helps apprehend security threats, prevalent on social media platforms. In such instances, language identification is an imperative task of code-mixed text. The focus of this paper is to carry out a word-level language identification (WLLI) of Malayalam-English code-mixed data, from social media platforms like YouTube. This study was centered around BERT, a transformer model, along with its variants -CamemBERT, DistilBERT -for intuitive perception of the language at the word-level. The propounded approach entails tagging Malayalam-English code-mixed data set with six labels: Malayalam (mal), English (eng), acronyms (acr), universal (univ), mixed (mix) and undefined (undef). Newly developed corpus of Malayalam-English was deployed for appraisal of the effectiveness of state-of-the-art models like BERT. Evaluation of the proffered approach, accomplished with other code-mixed language such as Hindi-English, notched a 9% increase in the F1-score.INDEX TERMS Natural language processing, language identification, bidirectional encoder representations from transformers (BERT), text mining, corpus preparation, deep learning, code-mixed

show abstract

Section: Related Workmentioning

confidence: 99%

Transformer Based Language Identification for Malayalam-English Code-Mixed Text

Thara

Poornachandran

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The evaluation approach is based on the selflearning approach [60]. The basic mechanism for extracting the contextual meaning is based on the condition that the left and right words to the pivot word must belong to two different languages [61][62][63]. The statistical approach based on set theory intersection concept is applied here in the proposal for annotating the words.…”

Section: Context Retrieval Evaluationmentioning

confidence: 99%

Hatred and trolling detection transliteration framework using hierarchical LSTM in code-mixed social media text

Shekhar

Garg

Agrawal

et al. 2021

Complex Intell. Syst.

Self Cite

105

View full text Add to dashboard Cite

The paper describes the usage of self-learning Hierarchical LSTM technique for classifying hatred and trolling contents in social media code-mixed data. The Hierarchical LSTM-based learning is a novel learning architecture inspired from the neural learning models. The proposed HLSTM model is trained to identify the hatred and trolling words available in social media contents. The proposed HLSTM systems model is equipped with self-learning and predicting mechanism for annotating hatred words in transliteration domain. The Hindi–English data are ordered into Hindi, English, and hatred labels for classification. The mechanism of word embedding and character-embedding features are used here for word representation in the sentence to detect hatred words. The method developed based on HLSTM model helps in recognizing the hatred word context by mining the intention of the user for using that word in the sentence. Wide experiments suggests that the HLSTM-based classification model gives the accuracy of 97.49% when evaluated against the standard parameters like BLSTM, CRF, LR, SVM, Random Forest and Decision Tree models especially when there are some hatred and trolling words in the social media data.

show abstract

“…In past, we witness magnitude of work to address standard code-mixing natural language understanding (NLU) tasks such as language identification (Shekhar et al, 2020;Singh et al, 2018a;Ramanarayanan et al, 2019), POS tagging (Singh et al, 2018b;Vyas et al, 2014), named entity recognition (Singh et al, 2018a), and dependency pars-ing (Zhang et al, 2019a) along with sentence classification tasks like sentiment analysis (Patwa et al, 2020;Joshi et al, 2016), stance detection (Utsav et al, 2020), and sarcasm detection (Swami et al, 2018). Unlike code-mixed NLU, natural language generation (NLG) of code-mixed text is highly understudied.…”

Section: Introductionmentioning

confidence: 99%

HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text

Srivastava¹,

Singh²

2021

Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

Text generation is a highly active area of research in the computational linguistic community. The evaluation of the generated text is a challenging task and multiple theories and metrics have been proposed over the years.Unfortunately, text generation and evaluation are relatively understudied due to the scarcity of high-quality resources in code-mixed languages where the words and phrases from multiple languages are mixed in a single utterance of text and speech. To address this challenge, we present a corpus (HinGE) for a widely popular code-mixed language Hinglish (code-mixing of Hindi and English languages). HinGE has Hinglish sentences generated by humans as well as two rule-based algorithms corresponding to the parallel Hindi-English sentences. In addition, we demonstrate the inefficacy of widely-used evaluation metrics on the code-mixed data. The HinGE dataset will facilitate the progress of natural language generation research in code-mixed languages.

show abstract

Language identification framework in code-mixed social media text based on quantum LSTM — the word belongs to which language?

Cited by 32 publications

References 39 publications

Transformer Based Language Identification for Malayalam-English Code-Mixed Text

Transformer Based Language Identification for Malayalam-English Code-Mixed Text

Hatred and trolling detection transliteration framework using hierarchical LSTM in code-mixed social media text

HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text

Contact Info

Product

Resources

About