Revisiting Hate Speech Benchmarks: From Data Curation to System Deployment

Abstract

Social media is awash with hateful content, much of which is often veiled with linguistic and topical diversity. The benchmark datasets used for hate speech detection do not account for such divagation as they are predominantly compiled using hate lexicons. However, capturing hate signals becomes challenging in neutrally-seeded malicious content. Thus, designing models and datasets that mimic the real-world variability of hate warrants further investigation.To this end, we present GOTHate, a large-scale code-mixed crowdsourced dataset of around 51𝑘 posts for hate speech detection from Twitter. GOTHate is neutrally seeded, encompassing different languages and topics. We conduct detailed comparisons of GOTHate with the existing hate speech datasets, highlighting its novelty. We benchmark it with 10 recent baselines. Our extensive empirical and benchmarking experiments suggest that GOTHate is hard to classify in a text-only setup. Thus, we investigate how adding endogenous signals enhances the hate speech detection task. We augment GOTHate with the user's timeline information and ego network, bringing the overall data source closer to the real-world setup for understanding hateful content. Our proposed solution HEN-mBERT, is a modular, multilingual, mixture-of-experts model that enriches the linguistic subspace with latent endogenous signals from history, topology, and exemplars. HEN-mBERT transcends the best baseline by 2.5% and 5% in overall macro-F1 and hate class F1, respectively. Inspired by our experiments, in partnership with Wipro AI, we are developing a semi-automated pipeline to detect hateful content as a part of their mission to tackle online harm. 1 CCS CONCEPTS• Computing methodologies → Language resources; Supervised learning by classification.

Proactively Reducing the Hate Intensity of Online Posts via Hate Speech Normalization

“…Along similar lines, we have hate and offense datasets from diverse online forums like Wikipedia [64], Stormfront [16], Facebook [46], Reddit [37] etc. On the other hand, to overcome the use of hate lexicons in curating the datasets, large-scale neutrally seeded datasets have also been proposed [19,28,55]. The initial research in hate speech datasets focused on multi-class text classification assuming English posts.…”