Language variation and algorithmic bias: understanding algorithmic bias in British English automatic speech recognition

Abstract

All language is characterised by variation which language users employ to construct complex social identities and express social meaning. Like other machine learning technologies, speech and language technologies (re)produce structural oppression when they perform worse for marginalised language communities. Using knowledge and theories from sociolinguistics, I explore why commercial automatic speech recognition systems and other language technologies perform significantly worse for already marginalised populations, such as second-language speakers and speakers of stigmatised varieties of English in the British Isles. Situating language technologies within the broader scholarship around algorithmic bias, consider the allocative and representational harms they can cause even (and perhaps especially) in systems which do not exhibit predictive bias, narrowly defined as differential performance between groups. This raises the question whether addressing or "fixing" this "bias" is actually always equivalent to mitigating the harms algorithmic systems can cause, in particular to marginalised communities. CCS CONCEPTS• Computing methodologies → Natural language processing; Speech recognition.

Incorporating automatic speech recognition methods into the transcription of police-suspect interviews: factors affecting automatic performance

“…Factors relating to a speaker's linguistic background, such as accent, can prove challenging for an automatic transcription system. Previous work has demonstrated that the performance of ASR systems declines significantly when confronted with speech that diverges from the "standard" variety; this has been found for non-native-accented speech in English (Meyer et al, 2020;DiChristofano et al, 2022;Markl, 2022) and Dutch (Feng et al, 2021), as well as for non-standard regionallyaccented speech in Brazilian Portuguese (Lima et al, 2019) and British English (Markl, 2022). Markl (2022) compared the performance of Google and Amazon transcription services across multiple accents of British English.…”

“…Many researchers have suggested that the composition of training datasets can cause bias within automatic systems (Tatman, 2017;Koenecke et al, 2020;Meyer et al, 2020;Feng et al, 2021) and that the underrepresentation of certain accents leads to a decline in performance for those varieties. Markl (2022) reports that certain substitution errors identified for speakers of nonstandard regional accents of British English suggest that there is an overrepresentation of Southern accents in the training data or that acoustic models are being trained only on more prestigious Southern varieties, such as Received Pronunciation. Similarly, Wassink et al (2022) claim that 20% of the errors within their data would be addressed by incorporating dialectal forms of ethnic varieties of American English (African American, ChicanX, and Native American) into the training of the automatic systems.…”

“…In recent years, a growing body of research has focused on systematic bias within automatic systems, i.e., underperformance for certain demographic groups, and significant disparities in performance have been demonstrated across accents. Transcripts tend to be significantly less accurate for non-native speakers (DiChristofano et al, 2022) or speakers of nonstandard regional varieties (Markl, 2022). However, a limitation of work in this area is the use of word error rate (WER) for evaluating performance.…”

Data Ethics Emergency Drill: A Toolbox for Discussing Responsible AI for Industry Teams

“…how do you model a heterogeneous everchanging group of people (2) the problem addressed a responsible AI value in their team (3) the problem appears to be caused by a realistic data ethics issue i.e. resulting from data or an algorithm using data (4) prompts are in the form of written information from people within the company (5) prompts increase in severity or difficulty over time (6) prompts have clear directives on what to discuss (7) prompts involve a variety of stakeholders This example is inspired by recent research that has found bias against second language speakers of English [38] in automatic speech recognition systems (ASR).…”

Identifying Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction

“…Accommodating technology shortcomings limits the potential benefits of technologies. However, when technologies with performance disparities are used in consequential domains -such as in job application videos -degraded service can not only stigmatize users but also lead to other types of harm, such as allocative harms [121].…”