Language Identification and Named Entity Recognition in Hinglish Code Mixed Tweets

Singh, Kunwarjeet; Sen, Indira; Kumaraguru, Ponnurangam

doi:10.18653/v1/p18-3008

Cited by 40 publications

(25 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Masked Language Modeling: We fine-tune mBERT on the masked language modeling objective, following Khanuja et al (2020b), on a combination of in-domain code-mixed movie scripts and publicly available datasets by Roy et al (2013) and Bhat et al (2018) to obtain modified mBERT (mod-mBERT) to be fine-tuned on the sentencepair classification task. Transliteration: We perform token-level language identification and transliterate the detected Romanized Hindi words in CS-NLI to Devanagari script using the approach in Singh et al (2018), to enable mBERT to better understand them. Translation: Due to the difficulty in training codemixed to monolingual translation models, we follow the approach in Dhar et al (2018) to obtain translations.…”

Section: Addressing Code-mixingmentioning

confidence: 99%

Detecting Entailment in Code-Mixed Hindi-English Conversations

Chakravarthy¹,

Umapathy²,

Black³

2020

Proceedings of the Sixth Workshop on Noisy User-Generated Text (W-Nut 2020)

View full text Add to dashboard Cite

The presence of large-scale corpora for Natural Language Inference (NLI) has spurred deep learning research in this area, though much of this research has focused solely on monolingual data. Code-mixing is the intertwined usage of multiple languages, and is commonly seen in informal conversations among polyglots. Given the rising importance of dialogue agents, it is imperative that they understand code-mixing, but the scarcity of code-mixed Natural Language Understanding (NLU) datasets has precluded research in this area. The dataset by Khanuja et al. (2020a) for detecting conversational entailment in codemixed Hindi-English text is the first of its kind. We investigate the effectiveness of language modeling, data augmentation, translation, and architectural approaches to address the codemixed, conversational, and low-resource aspects of this dataset. We obtain +8.09% test set accuracy over the current state of the art.

show abstract

Section: Addressing Code-mixingmentioning

confidence: 99%

Detecting Entailment in Code-Mixed Hindi-English Conversations

Chakravarthy¹,

Umapathy²,

Black³

2020

Proceedings of the Sixth Workshop on Noisy User-Generated Text (W-Nut 2020)

View full text Add to dashboard Cite

show abstract

“…We use our system to backtransliterate the Hindi English corpora from the LinCE 6 benchmark . The NER corpus is from Singh et al (2018a) and has 2,079 tweets while the POS tagging corpus is from Singh et al (2018b) and has 1,489 tweets. Some statistics about the datasets are presented in Table 7.…”

Section: Released Datasetsmentioning

confidence: 99%

Quasi Bidirectional Encoder Representations from Transformers for Word Sense Disambiguation

Bevilacqua¹,

Navigli²

2019

Proceedings - Natural Language Processing in a Deep Learning World

View full text Add to dashboard Cite

While contextualized embeddings have produced performance breakthroughs in many Natural Language Processing (NLP) tasks, Word Sense Disambiguation (WSD) has not benefited from them yet. In this paper, we introduce QBERT, a Transformerbased architecture for contextualized embeddings which makes use of a coattentive layer to produce more deeply bidirectional representations, better-fitting for the WSD task. As a result, we are able to train a WSD system that beats the state of the art on the concatenation of all evaluation datasets by over 3 points, also outperforming a comparable model using ELMo.

show abstract

“…Bhargava et al (2016) proposed an algorithm which uses a hybrid approach of a dictionary cum supervised classification approach for identifying entities in Code Mixed Text of Indian Languages such as Hindi-English and Tamil-English. Nelakuditi et al (2016) reported work on annotating code mixed English-Telugu data collected from social media site Facebook and creating automatic POS Taggers for this corpus, Singh et al (2018a) presented an exploration of automatic NER of Hindi-English code-mixed data, Singh et al (2018b) presented a corpus for NER in Hindi-English Code-Mixed along with experiments on their machine learning models. To the best of our knowledge the corpus we created is the first Telugu-English code-mixed corpus with named entity tags.…”

Section: Background and Related Workmentioning

confidence: 99%

Corpus Creation and Analysis for Named Entity Recognition in Telugu-English Code-Mixed Social Media Data

Srirangam¹,

Reddy²,

Singh

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop

View full text Add to dashboard Cite

Named Entity Recognition(NER) is one of the important tasks in Natural Language Processing(NLP) and also is a sub task of Information Extraction. In this paper we present our work on NER in Telugu-English code-mixed social media data. Code-Mixing, a progeny of multilingualism is a way in which multilingual people express themselves on social media by using linguistics units from different languages within a sentence or speech context. Entity Extraction from social media data such as tweets(twitter) 1 is in general difficult due to its informal nature, code-mixed data further complicates the problem due to its informal, unstructured and incomplete information. We present a Telugu-English code-mixed corpus with the corresponding named entity tags. The named entities used to tag data are Person('Per'), Organization('Org') and Location('Loc'). We experimented with the machine learning models Conditional Random Fields(CRFs), Decision Trees and Bidirectional LSTMs on our corpus which resulted in a F1-score of 0.96, 0.94 and 0.95 respectively.

show abstract

Language Identification and Named Entity Recognition in Hinglish Code Mixed Tweets

Cited by 40 publications

References 12 publications

Detecting Entailment in Code-Mixed Hindi-English Conversations

Detecting Entailment in Code-Mixed Hindi-English Conversations

Quasi Bidirectional Encoder Representations from Transformers for Word Sense Disambiguation

Corpus Creation and Analysis for Named Entity Recognition in Telugu-English Code-Mixed Social Media Data

Contact Info

Product

Resources

About