Unveiling Identity Biases in Toxicity Detection : A Game-Focused Dataset and Reactivity Analysis Approach

Abstract

Identity biases arise commonly from annotated datasets, can be propagated in language models and can cause further harm to marginal groups. Existing bias benchmarking datasets are mainly focused on gender or racial biases and are made to pinpoint which class the model is biased towards. They also are not designed for the gaming industry, a concern for models built for toxicity detection in videogames' chat. We propose a dataset and a method to highlight oversensitive terms using reactivity analysis and the model's performance. We test our dataset against ToxBuster, a language model developed by Ubisoft fine-tuned for toxicity detection on multiplayer videogame's written chat, and Perspective API. We find that these toxicity models often automatically tag terms related to a community's identity as toxic, which prevents members of already marginalized groups to make their presence known or have a mature / normal conversation. Through this process, we have generated an interesting list of terms that trigger the models to varying degrees, along with insights on establishing a baseline through human annotations.

A Systematic Review of Toxicity in Large Language Models: Definitions, Datasets, Detectors, Detoxification Methods and Challenges

“…Post-training assessment necessitates the exploration of various methodologies to analyse potential biases within the model. In Van Dorpe et al [204], a set of templates was devised to scrutinize the impact of protected group presence or absence in toxic detection. This analysis includes the calculation of a reactivity score, determined by assessing the average predictive difference across all sentence templates.…”

“…Despite the primary focus of this review not being centred on mitigating toxic detection bias, we have observed evaluation metrics [49,146,147], bias analysis methodologies [117,204,212], and bias mitigation techniques [45,88,96,112,134,139,209,221] that are cornerstones to the improvement of these models. Biases in toxic detection are not only embedded during the training phase but also inherent in the base models [136,147], resulting in the exacerbation of these biases and making it harder to assess and mitigate them after training.…”

“…These biases not only affect the prediction capabilities of the models by considering features that were learned during the training process but do not extrapolate to reality, directly affecting the end user. The repercussions of these biases may censor users based on the way they express language [103], their health condition [225], and their beliefs [40], and can also amplify these biases when used in mitigation processes [204]. Despite considerable work having been undertaken in this area, the predominant focus lies on identity-related bias [45], neglecting to acknowledge a broader framework wherein all biases are encompassed.…”