Handling Bias in Toxic Speech Detection: A Survey

Abstract

The massive growth of social media usage has witnessed a tsunami of online toxicity in teams of hate speech, abusive posts, cyberbullying, etc. Detecting online toxicity is challenging due to its inherent subjectivity. Factors such as the context of the speech, geography, socio-political climate, and background of the producers and consumers of the posts play a crucial role in determining if the content can be flagged as toxic. Adoption of automated toxicity detection models in production can lead to a sidelining of the various demographic and psychographic groups they aim to help in the first place. It has piqued researchers' interest in examining unintended biases and their mitigation. Due to the nascent and multi-faceted nature of the work, complete literature is chaotic in its terminologies, techniques, and findings. In this paper, we put together a systematic study to discuss the limitations and challenges of existing methods.We start by developing a taxonomy for categorising various unintended biases and a suite of evaluation metrics proposed to quantify such biases. We take a closer look at each proposed method for evaluating and mitigating bias in toxic speech detection. To examine the limitations of existing methods, we also conduct a case study to introduce the concept of bias shift due to knowledge-based bias mitigation methods. The survey concludes with an overview of the critical challenges, research gaps and future directions.While reducing toxicity on online platforms continues to be an active area of research, a systematic study of various biases and their mitigation strategies will help the research community produce robust and fair models.

Unveiling Identity Biases in Toxicity Detection : A Game-Focused Dataset and Reactivity Analysis Approach

“…Toxicity detection is inherently complex and subjective, with different definitions and interpretations among researchers (Garg et al, 2022;Kowert, 2020). Biases also vary across communities, influenced by culture, origin and socio-political context.…”

“…In this study, we define biases as "prejudice in favour of or against one thing, person, or group compared with another usually in a way that's considered to be unfair" (University of California). Natural language processing encompasses a wide range of types of biases, categorized by their sources or the type of harm they cause (Sap et al, 2019;Garg et al, 2022). Our focus lies specifically on lexical identity biases, which refer to biases conveyed by terms related to one's identity or characteristic (Zhou et al, 2021).…”

“…Biases can emerge during dataset creation when practitioners sample data, annotators label data based on personal understanding, culture, and experiences, and practitioners aggregate labels. In this study, we specifically focus on the issue of models overestimating the toxicity of terms associated with certain concepts, leading to problematic false positives and even false negatives (Dixon et al, 2018;Kiritchenko and Mohammad, 2018;Prabhakaran et al, 2019;Sap et al, 2019;Garg et al, 2022). Existing research has primarily focused on biases in toxicity detection without considering the specific use-case of in-game chat, despite the widespread presence of toxicity in that particular context.…”

Bias Detection for Customer Interaction Data: A Survey on Datasets, Methods, and Tools

“…The COMPAS dataset contains outcomes within 2 years of the decision, for over 10,000 criminal defendants in Broward County, Florida. The ''Stop, Question and Frisk'' database 5 contains data from NYPD officers' interactions with potential suspects of committing a crime. Features include locality-based information like time, street name, area code, etc.…”

“…Levitin [4] highlights that as data is collected by humans, they decide what to collect and what not. The objective for which the data is collected and its respective planning leads to wrong analysis/conclusions, e.g., which population/features to select and what to label, also called lexical bias [5]. At the learning stage, it is the bias that exists due to the transfer of bias in the model and how much it affects certain groups while proposing a generalised model that will work for all groups in the data [6].…”