Multilingual and Multi-Aspect Hate Speech Analysis

Ousidhoum, Nedjma; Lin, Zizheng; Zhang, Hongming; Song, Yangqiu; Yeung, Dit–Yan

doi:10.48550/arxiv.1908.11049

Cited by 11 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Annotators are seen to be self-consistent in highlighting terms and providing final judgments 94.5% of the time. If we measure inter-annotator agreement across annotators, however, we observe a Fleiss kappa of only 0.15, which is comparable to prior work [9,21,31,35]. Given task subjectivity, this is why self-consistency checks are so important for ensuring data quality.…”

Section: Dataset Properties and Evaluationsupporting

confidence: 70%

An Information Retrieval Approach to Building Datasets for Hate Speech Detection

Rahman¹,

Balakrishnan²,

Murthy³

et al. 2021

Preprint

View full text Add to dashboard Cite

Building a benchmark dataset for hate speech detection presents several challenges. Firstly, because hate speech is relatively raree.g., less than 3% of Twitter posts are hateful [14] -random sampling of tweets to annotate is inefficient in capturing hate speech. A common practice is to only annotate tweets containing known "hate words", but this risks yielding a biased benchmark that only partially captures the real-world phenomenon of interest. A second challenge is that definitions of hate speech tend to be highly variable and subjective. Annotators having diverse prior notions of hate speech may not only disagree with one another but also struggle to conform to specified labeling guidelines.Our key insight is that the rarity and subjectivity of hate speech are akin to that of relevance in information retrieval (IR) [38]. This connection suggests that well-established methodologies for creating IR test collections might also be usefully applied to create better benchmark datasets for hate speech detection. Firstly, to intelligently and efficiently select which tweets to annotate, we apply established IR techniques of pooling and active learning. Secondly, to improve both consistency and value of annotations, we apply task decomposition [52] and annotator rationale [26] techniques. Using the above techniques, we create and share a new benchmark dataset 1 for hate speech detection with broader coverage than prior datasets. We also show a dramatic drop in accuracy of existing detection models when tested on these broader forms of hate. Collected annotator rationales not only provide documented support for labeling decisions but also create exciting future work opportunities for dual-supervision [50] and/or explanation generation in modeling.Content Warning: We discuss hate speech and provide examples that might be disturbing to read. Please be forewarned.

show abstract

Section: Dataset Properties and Evaluationsupporting

confidence: 70%

An Information Retrieval Approach to Building Datasets for Hate Speech Detection

Rahman¹,

Balakrishnan²,

Murthy³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Nevertheless, with the advances in multilingual parsers and deep learning technology, together with increasing pressures from policy-makers to handle hate speech issues at local resources, non-English HS detection toolkits have seen a steady increase. The figure indicates that about 51% of all works in this field are performed on English dataset, with an increase of proportion of other languages as well where Arabic (13% ) [93,59,12,143], Turkish (6%) [143,104], Greek (4%) [143,6,136], Danish (5%) [106,143], Hindi (4%) [121,22,88], German (4% ) [72,120], Malayalam (3%) [130,109], Tamil (3%) [130,20], Chinese (1%) [138,139,155], Italian (2%) [116], Urdu (1%) [126,95,7], Russian(1%) [17], Bengali (1% ) [62,127,69], Korean (1%) [91], French (1%) [16,102,50], Indonesian (1%) [14], Portuguese (1%) [14], Spanish (1%) [56] and Polish (1%) [118] seem to dominate the rest of the languages in this field.…”

Section: Statistical Trends Of Resultsmentioning

confidence: 99%

A systematic review of Hate Speech automatic detection using Natural Language Processing

Saroar¹,

Oussalah²

2021

Preprint

View full text Add to dashboard Cite

With the multiplication of social media platforms, which offer anonymity, easy access and online community formation and online debate, the issue of hate speech detection and tracking becomes a growing challenge to society, individual, policy-makers and researchers. Despite efforts for leveraging automatic techniques for automatic detection and monitoring, their performances are still far from satisfactory, which constantly calls for future research on the issue. This paper provides a systematic review of literature in this field, with a focus on natural language processing and deep learning technologies, highlighting the terminology, processing pipeline, core methods employed, with a focal point on deep learning architecture. From a methodological perspective, we adopt PRISMA guideline of systematic review of the last 10 years literature from ACM Digital Library and Google Scholar. In the sequel, existing surveys, limitations, and future research directions are extensively discussed.

show abstract

“…In traditional hate speech tasks, the most common algorithms for the task are SVM, Random Forests, and Decision Trees (Fortuna and Nunes, 2018). There are other studies on hate speech detections, based on Linguistic methods, n-grams, word2vec (Nobata et al, 2016), and logistic regression (Ousidhoum et al, 2019), Convolutional Neural Networks and Explainable Artificial Intelligence (Hardage and Najafirad, 2020), and Network Methods (Fan et al, 2020). With the advent of Transformers (Vaswani et al, 2017) structure and transfer learning in AI, BERT, which stands for Bidirectional Encoder Representations from Transformers (Devlin et al, 2018), becomes one of the most popular models for hate speech detection, and an increasing number of studies have shown the dominant performance of BERT in terms of detection tasks.…”

Section: Related Workmentioning

confidence: 99%

Speech Detection Task Against Asian Hate: BERT the Central, While Data-Centric Studies the Crucial

Lian¹

2022

Preprint

View full text Add to dashboard Cite

With the epidemic continuing, hatred against Asians is intensifying in countries outside Asia, especially among the Chinese. Thus, there is an urgent need to detect and prevent hate speech towards Asians effectively. In this work, we first create COVID-HATE-2022, an annotated dataset that is an extension of the anti-Asian hate speech dataset on Twitter, including 2,035 annotated tweets fetched in early February 2022, which are labeled based on specific criteria, and we present the comprehensive collection of scenarios of hate and nonhate tweets in the dataset. Second, we finetune the BERT models based on the relevant datasets, and demonstrate strategies including 1) cleaning the hashtags, usernames being @, URLs, and emojis before the fine-tuning process, and 2) training with the data while validating with the "clean" data (and the opposite) are not effective for improving performance. Third, we investigate the performance of advanced fine-tuning strategies with 1) modelcentric approaches, such as discriminative finetuning, gradual unfreezing, and warmup steps, and 2) data-centric approaches, which incorporate data trimming and data augmenting, and show that both strategies generally improve the performance, while data-centric ones outperform the others, which demonstrate the feasibility and effectiveness of the data-centric approaches. 2 The hate crime data for 2020 is available on the FBI's Crime Data Explorer: https: //crime-data-explorer.app.cloud.gov 3 See https://www.who.int/docs/ default-source/coronaviruse/ covid19-stigma-guide.pdf

show abstract

Multilingual and Multi-Aspect Hate Speech Analysis

Cited by 11 publications

References 0 publications

An Information Retrieval Approach to Building Datasets for Hate Speech Detection

An Information Retrieval Approach to Building Datasets for Hate Speech Detection

A systematic review of Hate Speech automatic detection using Natural Language Processing

Speech Detection Task Against Asian Hate: BERT the Central, While Data-Centric Studies the Crucial

Contact Info

Product

Resources

About