Online Multilingual Hate Speech Detection: Experimenting with Hindi and English Social Media

Abstract

The last two decades have seen an exponential increase in the use of the Internet and social media, which has changed basic human interaction. This has led to many positive outcomes. At the same time, it has brought risks and harms. The volume of harmful content online, such as hate speech, is not manageable by humans. The interest in the academic community to investigate automated means for hate speech detection has increased. In this study, we analyse six publicly available datasets by combining them into a single homogeneous dataset. Having classified them into three classes, abusive, hateful or neither, we create a baseline model and improve model performance scores using various optimisation techniques. After attaining a competitive performance score, we create a tool that identifies and scores a page with an effective metric in near-real-time and uses the same feedback to re-train our model. We prove the competitive performance of our multilingual model in two languages, English and Hindi. This leads to comparable or superior performance to most monolingual models.

Towards multidomain and multilingual abusive language detection: a survey

“…Based on Table 4, most studies implemented transformerbased architecture to deal with abusive language detection in a cross-lingual setting. However, we also observe some works that exploited a traditional machine learning approach, such as logistic regression [6,10,135], linear support vector machines [92,94], and support vector machines [59]. They used multilingual language representation or simple translation tools (to translate the data training to the target languages) for the knowledge sharing between languages.…”

“…2021 [135] Multiple models Experimented with several models including a joint-learning architecture which allow the model to learn from source and target languages sequentially.…”

“…They used multilingual language representation or simple translation tools (to translate the data training to the target languages) for the knowledge sharing between languages. Some studies also exploited several neural-based models such as LSTM [29,92,94,135], Bi-LSTMs [29], and GRU [6,28]. The more recent works adopted several transformer-based architectures due to the availability of multilingual transformer models such as Multilingual BERT [1,6,48,92,100,132,135], RoBERTa [30,31], XLM [28,132], and XLM-RoBERTa [30,31,48,110].…”

“…Some studies also exploited several neural-based models such as LSTM [29,92,94,135], Bi-LSTMs [29], and GRU [6,28]. The more recent works adopted several transformer-based architectures due to the availability of multilingual transformer models such as Multilingual BERT [1,6,48,92,100,132,135], RoBERTa [30,31], XLM [28,132], and XLM-RoBERTa [30,31,48,110]. Interestingly, we also notice some works that proposed a multichannel architecture based on the multilingual BERT model [20,130], which allows the model to learn the task in several languages sequentially.…”

“…Meanwhile, most neural-based models were coupled by multilingual word embedding models, including Facebook MUSE (Multilingual FastText) [6,89,92,94] and Babylon Embeddings [89]. Finally, the transformer-based architectures exploited the multilingual pre-trained model trained on the very big corpus such as Multilingual BERT [1,6,48,92,100,132,135], RoBERTa [30,31], ULMFit [31], and the recent XLM-RoBERTa [30,31,48,110]. It is worth noting that we also discover that some features were introduced to complement the language representation, providing languageagnostic information for knowledge transfer such as a hatespecific lexicon (HurtLex) [29,92,94] and emotion features based on emoji presence [28].…”