Fairness in Language Models Beyond English: Gaps and Challenges

Abstract

With language models becoming increasingly ubiquitous, it has become essential to address their inequitable treatment of diverse demographic groups and factors. Most research on evaluating and mitigating fairness harms has been concentrated on English, while multilingual models and non-English languages have received comparatively little attention. This paper presents a survey of fairness in multilingual and non-English contexts, highlighting the shortcomings of current research and the difficulties faced by methods designed for English. We contend that the multitude of diverse cultures and languages across the world makes it infeasible to achieve comprehensive coverage in terms of constructing fairness datasets. Thus, the measurement and mitigation of biases must evolve beyond the current datasetdriven practices that are narrowly focused on specific dimensions and types of biases and, therefore, impossible to scale across languages and cultures.

Leveraging AI to improve evidence synthesis in conservation

“…As previously discussed, it is likely that documents written in English and from developed countries form the bulk of the training corpus -this may limit the nature of responses to specific queries and enhance existing biases. Therefore, there is an urgent need for culturally sensitive multi-lingual LLMs [40]. Moreover, in the current LLM landscape there is a lack of transparency around algorithm development and reporting related to decisions algorithms make during the review process.…”

Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)

“…While doing so, it is important to culturally contextualize NLP metrics and models. Instead of plainly translating English models into Bengali, Hindi, etc., we need to carefully consider the dimensions of fairness and types and sources of bias specific to that cultural context Ramesh et al, 2023). To address this gap, this paper proposes a methodology for developing culturally centered bias-evaluation datasets in NLP.…”

Toward Cultural Bias Evaluation Datasets: The Case of Bengali Gender, Religious, and National Identity

“…While doing so, it is important to culturally contextualize NLP metrics and models. Instead of plainly translating English models into Bengali, Hindi, etc., we need to carefully consider the dimensions of fairness and types and sources of bias specific to that cultural context (Malik et al, 2022;Ramesh et al, 2023). To address this gap, this paper proposes a methodology for developing culturally centered bias-evaluation datasets in NLP.…”