Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media

Abstract

The availability of large annotated corpora from social media and the development of powerful classification approaches have contributed in an unprecedented way to tackle the challenge of monitoring users’ opinions and sentiments in online social platforms across time. Such linguistic data are strongly affected by events and topic discourse, and this aspect is crucial when detecting phenomena such as hate speech, especially from a diachronic perspective. We address this challenge by focusing on a real case study: the “Contro l’odio” platform for monitoring hate speech against immigrants in the Italian Twittersphere. We explored the temporal robustness of a BERT model for Italian (AlBERTo), the current benchmark on non-diachronic detection settings. We tested different training strategies to evaluate how the classification performance is affected by adding more data temporally distant from the test set and hence potentially different in terms of topic and language use. Our analysis points out the limits that a supervised classification model encounters on data that are heavily influenced by events. Our results show how AlBERTo is highly sensitive to the temporal distance of the fine-tuning set. However, with an adequate time window, the performance increases, while requiring less annotated data than a traditional classifier.

Resources and benchmark corpora for hate speech detection: a systematic review

“…5. Recent studies are showing how the volatile nature of topics, especially on social media, can hinder the predictive capability of supervised models trained on data collected with particular keyword sets (Wiegand et al 2019), or in restricted time spans (Florio et al 2020).…”

“Contro L’Odio”: A Platform for Detecting, Monitoring and Visualizing Hate Speech against Immigrants in Italian Social Media

“…For example, if events concerning landings of illegal immigrants happened, we could easily find an increasing number of hate tweets compared to this category, probably with a completely new vocabulary and never found in the previous months' tweets. A deeper analysis providing some experimental evidence of such hypothesis can be found in Florio et al (2020).…”

“…The aim of this study is to discover and analyze the topics of discussion emerging from the data, and their diachronic behavior. Our main statistical tool is the polarized weirdness index (Florio et al 2020). This word-level measure is based on the weirdness index (Ahmad, Gillam, and Tostevin 1999), an intuitive and flexible technique which can be applied to several domain of knowledge, and text types.…”

“…As a privileged place for the expression of opinions, feelings and emotions, social media are particularly suitable for conveying not only generic expressions of offensiveness and hatred, but hate speech, which affects individuals or groups of people because of their belonging to a vulnerable category, that is as they are characterized by a particular race or ethnicity, religion, gender or sexual orientation. In the background there are the tensions manifested at social level in relation to events and situations, that act as triggers of hate speech, sometimes giving rise to harmful hate campaigns (Florio et al 2020). Considering that social media allows for a wide and rapid dissemination of messages, the extreme expressions of verbal violence and their proliferation on the network are gradually taking shape as mandatory social emergencies.…”

Exploring Multi-Task Multi-Lingual Learning of Transformer Models for Hate Speech and Offensive Speech Identification in Social Media

“…Recently, multitask learning has been used for improving performance on NLP tasks [18,44], especially social media information extraction tasks [20], and more simpler variants have been tried for hate speech identification in our recent works [27,29]. [10] investigated the usage of AlBERTo on monitoring hate speech against Italian on Twitter. Their results show that even though AlBERTo is sensitive to the fine tuning set, it's performance increases given enough training time.…”