Bias and comparison framework for abusive language datasets

Abstract

Recently, numerous datasets have been produced as research activities in the field of automatic detection of abusive language or hate speech have increased. A problem with this diversity is that they often differ, among other things, in context, platform, sampling process, collection strategy, and labeling schema. There have been surveys on these datasets, but they compare the datasets only superficially. Therefore, we developed a bias and comparison framework for abusive language datasets for their in-depth analysis and to provide a comparison of five English and six Arabic datasets. We make this framework available to researchers and data scientists who work with such datasets to be aware of the properties of the datasets and consider them in their work.

Toxic language detection: A systematic review of Arabic datasets

“…pose a significant concern as they diminish the generalizability of models and may increase the risk of developing discriminatory models against certain social categories. Our awareness is limited to a single study (Wich et al, 2022) that reviewed biases in Arabic toxic language datasets. Specifically, the study covered six Arabic datasets and five in English.…”

Preventing profiling for ethical fake news detection

“… Al Kuwatly, Wich, and Groh (2020) identified annotator bias based on several demographic characteristics such as age, first language, and education level that leads to biased abusive language and hate speech detectors. Lastly, Wich, Bauer, and Groh (2020) found a negative effect of political bias in hate speech detection models and later developed a framework to analyse and uncover inherent biases in abusive language datasets ( Wich, Eder, Al Kuwatly, & Groh, 2022 ). In this paper, we address the ethical principles of fairness and prevention of harm ( High-Level Expert Group on AI, 2019 ).…”