A Twitter Corpus for Hindi-English Code Mixed POS Tagging

Abstract

Code-mixing is a linguistic phenomenon where multiple languages are used in the same occurrence that is increasingly common in multilingual societies. Codemixed content on social media is also on the rise, prompting the need for tools to automatically understand such content. Automatic Parts-of-Speech (POS) tagging is an essential step in any Natural Language Processing (NLP) pipeline, but there is a lack of annotated data to train such models. In this work, we present a unique language tagged and POS-tagged dataset of code-mixed English-Hindi tweets related to five incidents in India that led to a lot of Twitter activity. Our dataset is unique in two dimensions: (i) it is larger than previous annotated datasets and (ii) it closely resembles typical real-world tweets. Additionally, we present a POS tagging model that is trained on this dataset to provide an example of how this dataset can be used. The model also shows the efficacy of our dataset in enabling the creation of codemixed social media POS taggers. 3 This dataset also comprises of 772 Facebook posts and 762 WhatsApp messages 4 https:

Quasi Bidirectional Encoder Representations from Transformers for Word Sense Disambiguation

“…We use our system to backtransliterate the Hindi English corpora from the LinCE 6 benchmark . The NER corpus is from Singh et al (2018a) and has 2,079 tweets while the POS tagging corpus is from Singh et al (2018b) and has 1,489 tweets. Some statistics about the datasets are presented in Table 7.…”

Benchmark Dataset for Propaganda Detection in Czech Newspaper Texts

“…Different from the previous approaches, Aguilar and Solorio (2020) use language identification to create a code-switching ELMo from English ELMo (Peters et al, 2018). Later they show the effectiveness of their CS-ELMo by achieving state-of-theart POS tagging results on a Hindi-English dataset (Singh et al, 2018). They also employ multi-task learning where their auxiliary task is language identification with a simplified LID tag set for LID, POS, and NER tagging.…”

Parallel Sentence Retrieval From Comparable Corpora for Biomedical Text Simplification

“…We evaluate our models on five downstream tasks in the LinCE Benchmark (Aguilar et al, 2020a). We choose three named entity recognition (NER) tasks, Hindi-English (HIN-ENG) , Spanish-English (SPA-ENG) (Aguilar et al, 2018) and Modern Standard Arabic (MSA-EA) (Aguilar et al, 2018), and two part-of-speech (POS) tagging tasks, Hindi-English (HIN-ENG) (Singh et al, 2018b) and Spanish-English (SPA-ENG) (Soto and Hirschberg, 2017). We apply Roman-to-Devanagari transliteration on the Hindi-English datasets since the multilingual models are trained with data using that form.…”