Evaluating Gender Bias in Hindi-English Machine Translation

Ramesh, Krithika; Gupta, Gauri Shanker; Singh, Sanjay

doi:10.18653/v1/2021.gebnlp-1.3

Cited by 7 publications

(8 citation statements)

References 20 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cho et al ( 2019) introduced translation gender bias index (TGBI) as a metric to measure bias in NMT systems using gender-neutral source language sentences, originally for the Korean language. Ramesh et al (2021) showed that the TGBI metric can be applied to Hindi too. They constructed seven sets (P 1 to P 7 ) of gender-neutral sentences in Hindi which included: formal (S1), impolite (S2), informal (S3), occupation (S4), negative (S5), polite (S6), and positive (S7) versions.…”

Section: Tgbi Evaluation Using Gender-neutral Sentencesmentioning

confidence: 99%

“…This problem also exists for HI-EN Machine Translation (Ramesh et al, 2021). When put to use, such systems can cause various harms (Savoldi et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

“…For instance, in Hindi, elements like pronouns, adjectives, and verbs are often inflected with gender. Nonetheless, prior studies in other source languages often utilize gender-neutral sentences (Cho et al, 2019;Ramesh et al, 2021) for bias evaluation. Yet, in practice, many sentences inherently possess gender information.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

Singh

2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

Neural Machine Translation (NMT) models are state-of-the-art for machine translation. However, these models are known to have various social biases, especially gender bias. Most of the work on evaluating gender bias in NMT has focused primarily on English as the source language. For source languages different from English, most of the studies use gender-neutral sentences to evaluate gender bias. However, practically, many sentences that we encounter do have gender information. Therefore, it makes more sense to evaluate for bias using such sentences. This allows us to determine if NMT models can identify the correct gender based on the grammatical gender cues in the source sentence rather than relying on biased correlations with, say, occupation terms. To demonstrate our point, in this work, we use Hindi as the source language and construct two sets of gender-specific sentences: OTSC-Hindi and WinoMT-Hindi that we use to evaluate different Hindi-English (HI-EN) NMT systems automatically for gender bias. Our work highlights the importance of considering the nature of language when designing such extrinsic bias evaluation datasets.

show abstract

Section: Tgbi Evaluation Using Gender-neutral Sentencesmentioning

confidence: 99%

“…This problem also exists for HI-EN Machine Translation (Ramesh et al, 2021). When put to use, such systems can cause various harms (Savoldi et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

Singh

2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

show abstract

“…The problem with this method is that it needs a labeled gender dataset beforehand to train the classifier. Recent work by Ramesh et al (2021) tries to find out bias in English-Hindi machine translation. They implement a modified version of the TGBI metric based on grammatical considerations for Hindi.…”

Section: Related Workmentioning

confidence: 99%

Mitigating Gender Stereotypes in Hindi and Marathi

Kirtane¹,

Anand²

2022

Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)

View full text Add to dashboard Cite

As the use of natural language processing increases in our day-to-day life, the need to address gender bias inherent in these systems also amplifies. This is because the inherent bias interferes with the semantic structure of the output of these systems while performing tasks in natural language processing. While research is being done in English to quantify and mitigate bias, debiasing methods in Indic Languages are either relatively nascent or absent for some Indic languages altogether. Most Indic languages are gendered, i.e., each noun is assigned a gender according to each language's rules of grammar. As a consequence, evaluation differs from what is done in English. This paper evaluates the gender stereotypes in Hindi and Marathi languages. The methodologies will differ from the ones in the English language because there are masculine and feminine counterparts in the case of some words. We create a dataset of neutral and gendered occupation words, emotion words and measure bias with the help of Embedding Coherence Test (ECT) and Relative Norm Distance (RND). We also attempt to mitigate this bias from the embeddings. Experiments show that our proposed debiasing techniques reduce gender bias in these languages.

show abstract

“…Cho et al [2019] expect TGBI to be a representative measure for inter-system comparison, especially if the gap between the systems is noticeable. Recently, Ramesh et al [2021] extend TGBI to Hindi. In general, this is a suitable method for applications where male default is the predominant risk.…”

Section: Causal Biasmentioning

confidence: 99%

A Survey on Gender Bias in Natural Language Processing

Stanczak¹,

Augenstein²

2021

Preprint

View full text Add to dashboard Cite

Language can be used as a means of reproducing and enforcing harmful stereotypes and biases and has been analysed as such in numerous research. In this paper, we present a survey of 304 papers on gender bias in natural language processing. We analyse definitions of gender and its categories within social sciences and connect them to formal definitions of gender bias in NLP research. We survey lexica and datasets applied in research on gender bias and then compare and contrast approaches to detecting and mitigating gender bias. We find that research on gender bias suffers from four core limitations. 1) Most research treats gender as a binary variable neglecting its fluidity and continuity. 2) Most of the work has been conducted in monolingual setups for English or other high-resource languages. 3) Despite a myriad of papers on gender bias in NLP methods, we find that most of the newly developed algorithms do not test their models for bias and disregard possible ethical considerations of their work. 4) Finally, methodologies developed in this line of research are fundamentally flawed covering very limited definitions of gender bias and lacking evaluation baselines and pipelines. We see overcoming these limitations as a necessary development in future research.

show abstract

Evaluating Gender Bias in Hindi-English Machine Translation

Cited by 7 publications

References 20 publications

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

Mitigating Gender Stereotypes in Hindi and Marathi

A Survey on Gender Bias in Natural Language Processing

Contact Info

Product

Resources

About