StereoSet: Measuring stereotypical bias in pretrained language models

Abstract

A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or African Americans are athletic. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. It is important to quantify to what extent these biases are present in them. Although this is a rapidly growing area of research, existing literature lacks in two important aspects: 1) they mainly evaluate bias of pretrained language models on a small set of artificial sentences, even though these models are trained on natural data; 2) current evaluations focus on measuring bias without considering the language modeling ability of a model, which could lead to misleading trust on a model even if it is a poor language model. We address both these problems. We present StereoSet, a large-scale natural English dataset to measure stereotypical biases in four domains: gender, profession, race, and religion. We contrast both stereotypical bias and language modeling ability of popular models like BERT, GPT2, ROBERTA, and XLNET. We show that these models exhibit strong stereotypical biases. Our data and code are available at https://stereoset. mit.edu.

Simple Entity-Centric Questions Challenge Dense Retrievers

“…In general, machine learning has the ability to amplify biases presented implicitly and explicitly in the training data. Models that we reference in our study are based on BERT, which has been shown to learn and exacerbate stereotypes during training (e.g., Kurita et al 2019, Tan and Celis 2019, Nadeem et al 2021. We further train these models on Wikidata triples, which again has the potential to amplify harmful and toxic biases.…”

Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP

“…For contextualized embeddings, similar methods to alleviate the issue of undesirable biases and toxicity have been proposed (Dev et al, 2020;Nangia et al, 2020;Nadeem et al, 2020;Krause et al, 2020;Kaneko and Bollegala, 2021a). For text generation, Gehman et al (2020) propose domain-adaptive pretraining on non-toxic corpora as outlined by Gururangan et al (2020) and consider plug and play language models (Dathathri et al, 2020).…”

How Reliable are Model Diagnostics?

“…One approach to examining the behaviour of language models like BERT is to examine how they rank certain representative examples above others. We use two contemporary datasets that measure how often stereotypes are ranked above antistereotypes -StereoSet (Nadeem et al, 2020) and CrowS-Pairs (Nangia et al, 2020). Both datasets measure ss = 100 StereoSet Nadeem et al (2020) propose a benchmark that contains intra-sentence and intersentence examples of stereotypes and antistereotypes.…”

“…Our findings demonstrate that model diagnostics can be unreliable on multiple fronts. To illustrate our point, we select three diagnostics tasks -StereoSet (Nadeem et al, 2020), CrowS-Pairs (Nangia et al, 2020), and SEATs (May et al, 2019) to base our empirical evaluation on. Overall, we find that likelihood-based and representationbased diagnostics measured multiple times on the same training setup can result in wildly different findings.…”