Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification

Abstract

Unintended bias in Machine Learning can manifest as systemic differences in performance for different demographic groups, potentially compounding existing challenges to fairness in society at large. In this paper, we introduce a suite of threshold-agnostic metrics that provide a nuanced view of this unintended bias, by considering the various ways that a classifier's score distribution can vary across designated groups. We also introduce a large new test set of online comments with crowd-sourced annotations for identity references. We use this to show how our metrics can be used to find new and potentially subtle unintended bias in existing public models. CCS CONCEPTS• Computing methodologies → Supervised learning by classification; Model verification and validation. ACM Reference Format:

Vindication, virtue, and vitriol

“…8 https ://www.bbc.co.uk/news/uk-north ern-irela nd-53386 976. 9 https ://www.thegu ardia n.com/uk-news/2020/may/14/polic e-vow-to-break -up-plann ed-anti-lockd ownprote sts-in-uk-citie s 10 https ://www.teleg raph.co.uk/news/2020/04/20/coron aviru s-world -erupt s-prote st-again st-lockd ownpictu res/ 11 https ://www.washi ngton post.com/world /europ e/face-masks -coron aviru s-uk/2020/07/14/d05df b7c-c5d4-11ea-a825-87220 04e41 50_story .html. 12 https ://www.teleg raph.co.uk/news/2020/04/06/brita ins-hubri stic-scien tific -advis ers-wrong -publi c-shoul d-weari ng/.…”

“…Data from Kaggle's 2012 challenge, "Detecting Insults in Social Commentary" [5], were used to evaluate the success of the approach, this being in keeping with our definition of abuse (many more recent corpora define this differently, e.g., "toxicity", as in the Jigsaw corpus [9], is much broader). Our approach was shown to have an accuracy of 80%, and a precision/recall/F1 of 0.72/0.47/0.57.…”

Pairwise Fairness for Ranking and Regression

“…With two protected groups, pinned AUC works by resampling the data such that each of the two groups make up 50% of the data, and then calculating the ROC AUC on the resampled dataset. Based on the wellknown equivalence between ROC AUC and average pairwise accuracy, Borkan et al (2019) demonstrate that pinned AUC, as well as their proposed weighted pinned AUC metric, can be decomposed as a linear combination of within-group and cross-group pairwise accuracies. In other words, both pinned AUC and weighted pinned AUC can be written as linear combinations of different pairwise accuracies A Gi>Gj in (1).…”

Studying Anti-Social Behaviour on Reddit with Communalytic

“…Prior literature on detecting anti-social acts at scale has primarily used supervised machine learning that predominantly relies on content-based features to identify relevant posts (Al-Makhadmeh & Tolba, 2019; Kwok & Wang, 2013;Pitsilis, Ramampiaro, & Langseth, 2018;Gorrell et al, 2019;Borkan et al, 2019;. For example, Dybala et al (2010) used support vector machines (SVM) to classify comments posted on unofficial school websites in Japan into those that are potentially harmful and not.…”