On Measuring Gender Bias in Translation of Gender-neutral Pronouns

Abstract

Ethics regarding social bias has recently thrown striking issues in natural language processing. Especially for gender-related topics, the need for a system that reduces the model bias has grown in areas such as image captioning, content recommendation, and automated employment. However, detection and evaluation of gender bias in the machine translation systems are not yet thoroughly investigated, for the task being cross-lingual and challenging to define. In this paper, we propose a scheme for making up a test set that evaluates the gender bias in a machine translation system, with Korean, a language with genderneutral pronouns. Three word/phrase sets are primarily constructed, each incorporating positive/negative expressions or occupations; all the terms are gender-independent or at least not biased to one side severely. Then, additional sentence lists are constructed concerning formality of the pronouns and politeness of the sentences. With the generated sentence set of size 4,236 in total, we evaluate gender bias in conventional machine translation systems utilizing the proposed measure, which is termed here as translation gender bias index (TGBI). The corpus and the code for evaluation is available on-line 1 .

UNQOVERing Stereotyping Biases via Underspecified Questions

“…Arguably this setting is more natural, as it better aligns with how systems are used in real life. Several notable examples are coreference resolution (Rudinger et al, 2018;Zhao et al, 2018;Kurita et al, 2019), machine translation (Stanovsky et al, 2019;Cho et al, 2019), textual entailment (Dev et al, 2020a), language generation (Sheng et al, 2019), or clinical classification (Zhang et al, 2020).…”

Evaluating Gender Bias in Hindi-English Machine Translation

“…There has been little work done for bias in language models for Hindi, and to the best of our knowledge, there has been no previous work that measures and analyses bias for MT of Hindi. Our approach uses two existing and broad frameworks for assessing bias in MT, including the Word Embedding Fairness Evaluation (Badilla et al, 2020) and the Translation Gender Bias Index (Cho et al, 2019) on Hindi-English MT systems. We modify some of the existing procedures within these metrics required for compatibility with Hindi grammar.…”

“…1. Construction of an equity evaluation corpus (EEC) (Kiritchenko and Mohammad, 2018) for Hindi of size 26370 utterances using 1558 sentiment words and 1100 occupations following the guidelines laid out in Cho et al (2019).…”

Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus

“…Previous studies accounting for MT systems' strengths and weaknesses in the translation of gender shed light on the problem but, at the same time, have limitations. On one hand, the existing evaluations focused on gender bias were largely conducted on challenge datasets, which are controlled artificial benchmarks that provide a limited perspective on the extent of the phenomenon and may force unreliable conclusions (Prates et al, 2018;Cho et al, 2019;Escudé Font and Costa-jussà, 2019;Stanovsky et al, 2019). On the other hand, the natural corpora built on conversational language that were used in few studies (Elaraby et al, 2018;Vanmassenhove et al, 2018) include only a restricted quantity of not isolated gender-expressing forms, thus not permitting either extensive or targeted evaluations.…”

“…Previous attempts to test the production of gender-aware automatic translations solely focused on MT, where a widespread approach involves the creation of challenge datasets focused on specific linguistic phenomena. Prates et al (2018) and Cho et al (2019) construct template sentences using occupational or sentiment words associated with a gender-neutral pronoun, to be translated into an English gender-specified one ([x] is a professor: he/she is a professor). Similarly, the Occupations Test (Escudé Font and Costajussà, 2019) and Wino MT (Stanovsky et al, 2019) cast human entities into proto-or anti-stereotypical gender associations via coreference linking (e.g.…”