Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages

Mandl, Thomas; Modha, Sandip; Shahi, Gautam Kishore; Madhu, Hiren; Satapara, Shrey; Majumder, Prasenjit; Schaefer, Johannes; Ranasinghe, Tharindu; Zampieri, Marcos; Nandini, Durgesh; Jaiswal, Amit

doi:10.48550/arxiv.2112.09301

Cited by 9 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We test XLM-T (Barbieri et al, 2021), an XLM-R model (Conneau et al, 2020) pre-trained on an additional 198 million Twitter posts in over 30 languages. 7 XLM-R is a widely-used architecture for multilingual language modelling, which has been shown to achieve near state-of-the-art performance on multilingual hate speech detection (Banerjee et al, 2021;Mandl et al, 2021). We chose XLM-T over XLM-R after initial experiments showed the former to outperform the latter on several hate speech detection datasets as well as MHC.…”

Section: Multilingual Transformer Modelsmentioning

confidence: 99%

HateCheck: Functional Tests for Hate Speech Detection Models

Röttger¹,

Vidgen²,

Nguyen³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HATECHECK, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HATECHECK's utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses.

show abstract

Section: Multilingual Transformer Modelsmentioning

confidence: 99%

HateCheck: Functional Tests for Hate Speech Detection Models

Röttger¹,

Vidgen²,

Nguyen³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…This third edition of HASOC Mandl et al [36] provided another set of tweets dataset with the same subtasks as HASOC 2020. The English dataset consists of 3843 training samples and 1281 samples in the test set.…”

Section: Hasoc 2021mentioning

confidence: 99%

T5 for Hate Speech, Augmented Data, and Ensemble

Adewumi,

Sabry,

Abid

et al. 2023

Sci

View full text Add to dashboard Cite

We conduct relatively extensive investigations of automatic hate speech (HS) detection using different State-of-The-Art (SoTA) baselines across 11 subtasks spanning six different datasets. Our motivation is to determine which of the recent SoTA models is best for automatic hate speech detection and what advantage methods, such as data augmentation and ensemble, may have on the best model, if any. We carry out six cross-task investigations. We achieve new SoTA results on two subtasks—macro F1 scores of 91.73% and 53.21% for subtasks A and B of the HASOC 2020 dataset, surpassing previous SoTA scores of 51.52% and 26.52%, respectively. We achieve near-SoTA results on two others—macro F1 scores of 81.66% for subtask A of the OLID 2019 and 82.54% for subtask A of the HASOC 2021, in comparison to SoTA results of 82.9% and 83.05%, respectively. We perform error analysis and use two eXplainable Artificial Intelligence (XAI) algorithms (Integrated Gradient (IG) and SHapley Additive exPlanations (SHAP)) to reveal how two of the models (Bi-Directional Long Short-Term Memory Network (Bi-LSTM) and Text-to-Text-Transfer Transformer (T5)) make the predictions they do by using examples. Other contributions of this work are: (1) the introduction of a simple, novel mechanism for correcting Out-of-Class (OoC) predictions in T5, (2) a detailed description of the data augmentation methods, and (3) the revelation of the poor data annotations in the HASOC 2021 dataset by using several examples and XAI (buttressing the need for better quality control). We publicly release our model checkpoints and codes to foster transparency.

show abstract

“…The HASOC 2021 Marathi dataset is a dataset presented by the Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages (HASOC 2021) track (Mandl et al, 2021) in the Forum for Information Retrieval Evaluation (FIRE 2021). It contains a total of 2,499 tweets in Marathi manually annotated by native speakers of the language.…”

Section: Downstream Evaluationmentioning

confidence: 99%

Spread Love Not Hate: Undermining the Importance of Hateful Pre-training for Hate Speech Detection

Shantanu¹,

Omkar²,

Aditya³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-training large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. Although this method has proven to be effective for many domains, it might not always provide desirable benefits. In this paper, we study the effects of hateful pre-training on low-resource hate speech classification tasks. While previous studies on the English language have emphasized its importance, we aim to augment their observations with some non-obvious insights. We evaluate different variations of tweet-based BERT models pre-trained on hateful, non-hateful, and mixed subsets of a 40M tweet dataset. This evaluation is carried out for the Indian languages Hindi and Marathi. This paper is empirical evidence that hateful pre-training is not the best pre-training option for hate speech detection. We show that pre-training on non-hateful text from the target domain provides similar or better results. Further, we introduce HindTweetBERT and MahaTweetBERT, the first publicly available BERT models pre-trained on Hindi and Marathi tweets, respectively. We show that they provide state-of-the-art performance on hate speech classification tasks. We also release hateful BERT for the two languages and a gold hate speech evaluation benchmark HateEval-Hi and HateEval-Mr consisting of manually labeled 2000 tweets each. The models and data are available at https://github.com/l3cube-pune/MarathiNLP .

show abstract

Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages

Cited by 9 publications

References 0 publications

HateCheck: Functional Tests for Hate Speech Detection Models

HateCheck: Functional Tests for Hate Speech Detection Models

T5 for Hate Speech, Augmented Data, and Ensemble

Spread Love Not Hate: Undermining the Importance of Hateful Pre-training for Hate Speech Detection

Contact Info

Product

Resources

About