Assessing gender bias in machine translation: a case study with Google Translate

Abstract

Recently there has been a growing concern in academia, industrial research labs and the mainstream commercial media about the phenomenon dubbed as machine bias, where trained statistical models -unbeknownst to their creators -grow to reflect controversial societal asymmetries, such as gender or racial bias. A significant number of Artificial Intelligence tools have recently been suggested to be harmfully biased towards some minority, with reports of racist criminal behavior predictors, Apple's Iphone X failing to differentiate between two distinct Asian people and the now infamous case of Google photos' mistakenly classifying black people as gorillas. Although a systematic study of such biases can be difficult, we believe that automated translation tools can be exploited through gender neutral languages to yield a window into the phenomenon of gender bias in AI.In this paper, we start with a comprehensive list of job positions from the U.S. Bureau of Labor Statistics (BLS) and used it in order to build sentences in constructions like "He/She is an Engineer" (where "Engineer" is replaced by the job position of interest) in 12 different gender neutral languages such as Hungarian, Chinese, Yoruba, and several others. We translate these sentences into English using the Google Translate API, and collect statistics about the frequency of female, male and gender-neutral pronouns in the translated output. We then show that Google Translate exhibits a strong tendency towards male defaults, in particular for fields typically associated to unbalanced gender distribution or stereotypes such as STEM (Science, Technology, Engineering and Mathematics) jobs. We ran these statistics against BLS' data for the frequency of female participation in each job position, in which we show that Google Translate fails to reproduce a real-world distribution of female workers. In summary, we provide experimental evidence that even if one does not expect in principle a 50:50 pronominal gender distribution, Google Translate yields male defaults much more frequently than what would be expected from demographic data alone.We believe that our study can shed further light on the phenomenon of machine bias and are hopeful that it will ignite a debate about the need to augment current statistical translation tools with debiasing techniques -which can already be found in the scientific literature.

The practical ethics of bias reduction in machine translation: why domain adaptation is better than data debiasing

“…Prates et al (2019) have argued that, given well-attested gender asymmetries in society, a 50:50 pronominal split in NMT outputs is unrealistic when sentences in gender-neutral languages (e.g., Finnish, Hungarian, Turkish) are translated into gendered languages (e.g., English, French, German). Nonetheless, they provide experimental evidence which suggests that Google Translate yields masculine defaults much more frequently than would be expected from demographic data alone (Prates et al 2019). Other researchers have developed techniques for mitigating biases in monolingual English NLP tools, with a handful of techniques applied to the more complex problem of inflected languages.…”

Frontiers in Service Science: The Management of Data Analytics Services: New Challenges and Future Directions

“…Many languages such as Hungarian or Chinese are gender neutral. Yet Google Translate exhibited a strong bias toward male when translating certain sentences in English (such as "he is a doctor"; Prates et al 2020). Google has since successfully updated its system to provide both feminine and masculine translations (Johnson 2017).…”

Conceptualisations of landscape differ across European languages

“…Cross-cultural work often uses translations of English lexicons (e.g., to evaluate scenic quality of landscape objects across countries) implicitly promoting the idea that meanings encoded in a lexicon are universal [4]. Such approaches are becoming more common with the availability of machine translation algorithms based on large-scale text analysis [5][6][7][8][9][10]. However, by assuming one-to-one translatability of words, these approaches do not capture cross-linguistic variability in word meaning which has been shown to exist for a variety of domains-even when comparing closely related languages [11,12].…”

“…However, by assuming one-to-one translatability of words, these approaches do not capture cross-linguistic variability in word meaning which has been shown to exist for a variety of domains-even when comparing closely related languages [11,12]. Moreover, machine translation algorithms have also been shown to introduce new biases present in the original corpora, for example assigning gender to nouns in stereotypical ways to languages where gendering is not typical [6,13]. In the case of landscape, this could mean that specific cultural ways of thinking about landscape are imposed upon other cultures.…”