An effective way of word-level language identification for code-mixed facebook comments using word-embedding via character-embedding

Veena, Pamalla; Kumar, M. Anand; Soman, K. P.

doi:10.1109/icacci.2017.8126062

Cited by 13 publications

(12 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A Hindi-English Sentiment Analysis system for Twitter data to forecast the sentiment present in the data has been proposed in [21]. Researchers have used tf-idf vector and GloVe Vector features along with the Support Vector Regression (SVR) model.…”

Section: A Support Vector Machinementioning

confidence: 99%

Machine Learning Techniques for Sentiment Analysis of Code-Mixed and Switched Indian Social Media Text Corpus - A Comprehensive Review

Ahmad¹,

Singla²,

Anis³

et al. 2022

IJACSA

View full text Add to dashboard Cite

A comprehensive review of sentiment analysis for code-mixed and switched text corpus of Indian social media using machine learning (ML) approaches, based on recent research studies has been presented in this paper. Code-mixing and switching are linguistic behavior shown by the bilingual/multilingual population, primarily in spoken but also in written communication, especially on social media. Code-mixing involves combining lower linguistic units like words and phrases of a language into the sentences of other language (the base language) and code-switching involves switching to another language, for the length of one sentence or more. In code-mixing and switching, a bilingual person takes one or more words or phrases from one language and introduces them into another language while communicating in that language in spoken or written mode. People nowadays express their views and opinions on several issues on social media. In multilingual countries, people express their views using English as well as their native languages. Several reasons can be attributed to code-mixing. Lack of knowledge in one language on a particular subject, being empathetic, interjection and clarification are some to name. Sentiment analysis of monolingual social media content has been carried out for the last two decades. However, during recent years, Natural Language Processing (NLP) research focus has also shifted towards the exploration of code-mixed data, thereby, making code mixed sentiment analysis an evolving field of research. Systems have been developed using ML techniques to predict the polarity of code-mixed text corpus and to fine tune the existing models to improve their performance.

show abstract

Section: A Support Vector Machinementioning

confidence: 99%

Machine Learning Techniques for Sentiment Analysis of Code-Mixed and Switched Indian Social Media Text Corpus - A Comprehensive Review

Ahmad¹,

Singla²,

Anis³

et al. 2022

IJACSA

View full text Add to dashboard Cite

show abstract

“…SVM was the most widely used technique by researchers in language identification tasks. SVM is often implemented due to its capability to build an efficient classifier model and produce good performance [47]. From the selected studies, SVM has shown impressive performance.…”

Section: 1) Machine Learning Approachmentioning

confidence: 99%

“…From the selected studies, SVM has shown impressive performance. Veena et al [47] utilised a linear kernel SVM classifier and could achieve an accuracy of 93% for word-level Malayalam-English and 95% for Tamil-English code-mixed LID. Chaitanya et al [48] incorporated several machine learning methods with Word2Vec embedding for Hindi-English.…”

Section: 1) Machine Learning Approachmentioning

confidence: 99%

“…The capitalisation feature identifies capitalised initial letters and whether they occurred at the beginning of a sentence. Veena et al [47] stated that an adequate number of embedding data could be sufficiently applied to develop word-level features for code-mixed LID. Additionally, a few studies observed that the character Ngram presented satisfactory performance to solve LID for code-mixed text [53,56].…”

Section: 1) N-grammentioning

confidence: 99%

See 1 more Smart Citation

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

et al. 2022

View full text Add to dashboard Cite

The mix of native language with other languages (code-mixing) in social media has posed a severe challenge for language identification (LID) systems. It has encouraged research on code-mixed LID solutions. This study investigated the techniques, challenges, and dataset availability with corresponding quality criteria and developed a comprehensive framework for code-mixed LID. This study addressed four research issues to identify gaps and future work opportunities in tackling code-mixed LID challenges. Based on our analysis of reviewed studies, we outlined key points for future research in code-mixed LID. We demonstrated a taxonomy of applied techniques for code-mixed LID and highlighted the different technique variants. In code-mixed LID tasks, we discovered four significant challenges: ambiguity, lexical borrowing, non-standard words, and intra-word code-mixing. This systematic literature review recognised 32 code-mixed datasets available for LID. We proposed five features to describe the quality criteria dataset. The features are the number of instances or sentences, percentage of code-mixed types in the data, number of tokens, number of unique tokens, and average sentence length. Finally, we synthesised the methodologies and proposed a conceptual framework for subsequent studies through our literature analysis.

show abstract

“…Researchers gave importance to a wide range of machine learning algorithms using the Bayesian approach (BN) [24], maximum entropy classifier (MaxEnt) [25], multinomial Naive Bayes classifier (MNB), conditional random field (CRF) [26], support vector machines (SVM) classifier [27], and decision trees (DT). Some of the recent studies have presented unbiased measures of code-switching from microblogging sites, which is helpful for comparison among various types of code-mixed corpora [13], [28], [29].…”

Section: Related Workmentioning

confidence: 99%

Transformer Based Language Identification for Malayalam-English Code-Mixed Text

Thara

Poornachandran

2021

IEEE Access

View full text Add to dashboard Cite

Social media users have the proclivity to write majority of the data for under resourced languages in code-mixed format. Code-mixing is defined as mixing of two or more languages in a single sentence. Research in code-mixed text helps apprehend security threats, prevalent on social media platforms. In such instances, language identification is an imperative task of code-mixed text. The focus of this paper is to carry out a word-level language identification (WLLI) of Malayalam-English code-mixed data, from social media platforms like YouTube. This study was centered around BERT, a transformer model, along with its variants -CamemBERT, DistilBERT -for intuitive perception of the language at the word-level. The propounded approach entails tagging Malayalam-English code-mixed data set with six labels: Malayalam (mal), English (eng), acronyms (acr), universal (univ), mixed (mix) and undefined (undef). Newly developed corpus of Malayalam-English was deployed for appraisal of the effectiveness of state-of-the-art models like BERT. Evaluation of the proffered approach, accomplished with other code-mixed language such as Hindi-English, notched a 9% increase in the F1-score.INDEX TERMS Natural language processing, language identification, bidirectional encoder representations from transformers (BERT), text mining, corpus preparation, deep learning, code-mixed

show abstract

An effective way of word-level language identification for code-mixed facebook comments using word-embedding via character-embedding

Cited by 13 publications

References 6 publications

Machine Learning Techniques for Sentiment Analysis of Code-Mixed and Switched Indian Social Media Text Corpus - A Comprehensive Review

Machine Learning Techniques for Sentiment Analysis of Code-Mixed and Switched Indian Social Media Text Corpus - A Comprehensive Review

A Systematic Review on Language Identification of Code-Mixed Text: Techniques, Data Availability, Challenges, and Framework Development

Transformer Based Language Identification for Malayalam-English Code-Mixed Text

Contact Info

Product

Resources

About