Part-of-speech Tagging of Code-Mixed Social Media Text

Abstract

A common step in the processing of any text is the part-of-speech tagging of the input text. In this paper, we present an approach to tackle code-mixed text from three different languages Bengali, Hindi, and Tamilapart from English. Our system uses Conditional Random Field, a sequence learning method, which is useful to capture patterns of sequences containing code switching to tag each word with accurate part-of-speech information. We have used various pre-processing and post-processing modules to improve the performance of our system. The results were satisfactory, with a highest of 75.22% accuracy in Bengali-English mixed data. The methodology that we employed in the task can be used for any resource poor language. We adapted standard learning approaches that work well with scarce data. We have also ensured that the system is portable to different platforms and languages and can be deployed for real-time analysis.

Word Embeddings for Code-Mixed Language Processing

“…Our choice of tasks is primarily motivated by the availability of annotated CM data. There has been prior work on CM sentiment identification (Vilares and Alonso, 2016;Rudra et al, 2016; and POS tagging (Solorio and Liu, 2008;AlGhamdi et al, 2016;Ghosh et al, 2016). But we are not aware of any work that utilizes pre-trained bilingual embeddings for these tasks.…”

Development of knowledge­oriented system of machine translation based on the analytic­synthetic text processing

“…The work on studying the types of conceptual structures and their language correspondences started within the framework of exploring artificial intelligence [6,7]. The result of this work was the model of "conceptual dependences".…”

Part-of-Speech Tagging for Code-Switched, Transliterated Texts without Explicit Language Identification

“…As is typically the case in NLP, such pipelines suffer from the problem of cascading errors; e.g., failures of the language identification will cause problems in the tag prediction (Barman et al, 2016). Other approaches have trained supervised models on POS-annotated, code-switched data (Jamatia et al, 2015;Ghosh et al, 2016;Gupta et al, 2017;Barman et al, 2016;Sequiera et al, 2015, inter alia), resources which are expensive to create and unavailable for most language pairs.…”