Investigating the Impact of Gender Representation in ASR Training Data: a Case Study on Librispeech

Garnerin, Mahault; Rossato, Solange; Besacier, Laurent

doi:10.18653/v1/2021.gebnlp-1.10

Cited by 16 publications

(9 citation statements)

References 18 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We reviewed the techniques on identifying and resolving representation bias mostly in tabular data data sets. The existing research has briefly investigated these issues in other data types such as multimedia [13,26,71], text [33,49], graphs, streams [25], spatio-temporal [28], etc. Still, identification and resolving biases in visual data sets has drawn more attention from different research communities and in this section we present a review of the existing works.…”

Section: Expanding the Scope To Other Data Typesmentioning

confidence: 99%

Representation Bias in Data: A Survey on Identification and Resolution Techniques

Shahbazi¹,

Li²,

Asudeh³

et al. 2022

Preprint

View full text Add to dashboard Cite

The grand goal of data-driven decision-making is to help humans make decisions, not only easily and at scale but also wisely, accurately, and just. However, data-driven algorithms are only as good as the data they work with, while data sets, especially social data, often miss representing minorities. Representation Bias in data can happen due to various reasons ranging from historical discrimination to selection and sampling biases in the data acquisition and preparation methods. One cannot expect AI-based societal solutions to have equitable outcomes without addressing the representation bias. This paper surveys the existing literature on representation bias in the data. It presents a taxonomy to categorize the studied techniques based on multiple design dimensions and provide a side-by-side comparison of their properties. There is still a long way to fully address representation bias issues in data. The authors hope that this survey motivates researchers to approach these challenges in the future by observing existing work within their respective domains.

show abstract

Section: Expanding the Scope To Other Data Typesmentioning

confidence: 99%

Representation Bias in Data: A Survey on Identification and Resolution Techniques

Shahbazi¹,

Li²,

Asudeh³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In recent years, a new research area has emerged that investigates the discriminatory performance of AI systems and its causes (Hovy and Spruit, 2016;Garnerin et al, 2021). In the ASR field, traditional metrics like the aggregated WER and CER are used to measure the overall performance of the models.…”

Section: Analysis Of Asr Accuracy Wrt Speaker Metadatamentioning

confidence: 99%

Lahjoita puhetta -- a large-scale corpus of spoken Finnish with some benchmarks

Moisio¹,

Porjazovski²,

Rouhe³

et al. 2022

Preprint

View full text Add to dashboard Cite

The Donate Speech campaign has so far succeeded in gathering approximately 3600 hours of ordinary, colloquial Finnish speech into the Lahjoita puhetta (Donate Speech) corpus. The corpus includes over twenty thousand speakers from all the regions of Finland and from all age brackets. The primary goals of the collection were to create a representative, large-scale resource to study spontaneous spoken Finnish and to accelerate the development of language technology and speech-based services. In this paper, we present the collection process and the collected corpus, and showcase its versatility through multiple use cases. The evaluated use cases include: automatic speech recognition of spontaneous speech, detection of age, gender, dialect and topic and metadata analysis. We provide benchmarks for the use cases, as well downloadable, trained baseline systems with open-source code for reproducibility. One further use case is to verify the metadata and transcripts given in this corpus itself, and to suggest artificial metadata and transcripts for the part of the corpus where it is missing.

show abstract

“…Data balancing [7] [23], [24] [13], [25], [26] AT [27], [28], [29] [14], [30], [31], [32], [33], [34] MTL [8] [35], [36], [37], [38]…”

Section: Asv Asr Other ML Domainsmentioning

confidence: 99%

“…Feng et al [71] have analyzed the biases in a Dutch ASR system with respect to gender, age etc. Evaluations of ASR systems using criterion commonly used in Fair-ML research have been explored extensively [24,[72][73][74]. However, a systematic evaluation of fairness in ASV systems is scarce in current literature.…”

Section: Fairness In Asvmentioning

confidence: 99%

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Peri¹,

Somandepalli²,

Narayanan³

2022

Preprint

View full text Add to dashboard Cite

Speaker recognition is increasingly used in several everyday applications including smart speakers, customer care centers and other speech-driven analytics. It is crucial to accurately evaluate and mitigate biases present in machine learning (ML) based speech technologies, such as speaker recognition, to ensure their inclusive adoption. ML fairness studies with respect to various demographic factors in modern speaker recognition systems are lagging compared to other human-centered applications such as face recognition. Existing studies on fairness in speaker recognition systems are largely limited to evaluating biases at specific operating points of the systems, which can lead to false expectations of fairness. Moreover, there are only a handful of bias mitigation strategies developed for speaker recognition systems. In this paper, we systematically evaluate the biases present in speaker recognition systems with respect to gender across a range of system operating points. We also propose adversarial and multi-task learning techniques to improve the fairness of these systems. We show through quantitative and qualitative evaluations that the proposed methods improve the fairness of ASV systems over baseline methods trained using data balancing techniques. We also present a fairness-utility trade-off analysis to jointly examine fairness and the overall system performance. We show that although systems trained using adversarial techniques improve fairness, they are prone to reduced utility. On the other hand, multi-task methods can improve the fairness while retaining the utility. These findings can inform the choice of bias mitigation strategies in the field of speaker recognition.

show abstract

Investigating the Impact of Gender Representation in ASR Training Data: a Case Study on Librispeech

Cited by 16 publications

References 18 publications

Representation Bias in Data: A Survey on Identification and Resolution Techniques

Representation Bias in Data: A Survey on Identification and Resolution Techniques

Lahjoita puhetta -- a large-scale corpus of spoken Finnish with some benchmarks

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Contact Info

Product

Resources

About