Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias

Abstract

The one-sided focus on English in previous studies of gender bias in NLP misses out on opportunities in other languages: English challenge datasets such as GAP and Wino-Gender highlight model preferences that are "hallucinatory", e.g., disambiguating genderambiguous occurrences of 'doctor' as male doctors. We show that for languages with type B reflexivization, e.g., Swedish and Russian, we can construct multi-task challenge datasets for detecting gender bias that lead to unambiguously wrong model predictions: In these languages, the direct translation of 'the doctor removed his mask' is not ambiguous between a coreferential reading and a disjoint reading. Instead, the coreferential reading requires a non-gendered pronoun, and the gendered, possessive pronouns are anti-reflexive. We present a multilingual, multi-task challenge dataset, which spans four languages and four NLP tasks and focuses only on this phenomenon. We find evidence for gender bias across all task-language combinations and correlate model bias with national labor market statistics.

Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation

“…Several works created synthetic datasets to evaluate gender bias (Kiritchenko and Mohammad, 2018;González et al, 2020;Renduchintala and Williams, 2021), e.g., in the context of coreference (Rudinger et al, 2017;Zhao et al, 2018) and machine translation (Stanovsky et al, 2019;Prates et al, 2019;Kocmi et al, 2020), and some works used synthetic datasets to debias models (Saunders et al, 2020;Zhao et al, 2018). Webster et al (2018) and Gonen and Webster (2020), collected natural medium-scale (4.4K sentences) datasets from Wikipedia and reddit, re-spectively, and use them to evaluate gender bias in models of coreference resolution and machine translation.…”

Resources and Evaluations for Danish Entity Resolution

“…Danish has gendered possessive pronouns, but nongendered reflexive pronouns. This has made it useful as an unambiguous testbed for gender bias in natural language inference models, machine translation models, and language models (González et al, 2020). But automatic coreference resolution for Danish has received no attention, and there was no established evaluation set for this task.…”

Gender Inequality in Education in China

“…With the rise in the economy comes more slots in schools, but still, the money is needed, and with the economy rising, some families can't afford to pay the fees for all their children. Reports show that even though most families can send their children to elementary, primary and junior high school, they cannot send them to tertiary schools (González, et al, 2020). Another factor affecting gender inequality in education is the number of children in the household.…”