Improving Fairness in Speaker Recognition

Fenu, Gianni; Medda, Giacomo; Marras, Mirko; Meloni, Giacomo

doi:10.1145/3393822.3432325

Cited by 15 publications

(26 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data balancing [7] [23], [24] [13], [25], [26] AT [27], [28], [29] [14], [30], [31], [32], [33], [34] MTL [8] [35], [36], [37], [38]…”

Section: Asv Asr Other ML Domainsmentioning

confidence: 99%

“…As noted by Drozdowski et al [76], a majority of bias detection and mitigation works in biometrics focus on face recognition [25,36,54,56], and some in fingerprint matching [77,78]. Fairness in voice-based biometrics remains to be an under-explored field with only a handful of works [7][8][9]79].…”

Section: Fairness In Asvmentioning

confidence: 99%

“…Thus, it is crucial to consider biases that arise in different applications where ASV systems are deployed. While tremendous strides have been made in evaluating and improving fairness in applied ML fields [1], techniques for bias mitigation in ASV systems are limited [7,8]. Notably, most current bias evaluation frameworks for ASV are restricted to specific operating points of the systems [9,10], limiting their applicability to specific use-cases.…”

Section: Introductionmentioning

confidence: 99%

“…Bias mitigation strategies in ML systems often involve models trained using data balancing [11][12][13]. In the context of fairness in ASV systems, data balancing methods were studied with respect to age and gender [7]. However, it is not evident if such techniques are the most suitable to induce fairness.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Peri¹,

Somandepalli²,

Narayanan³

2022

Preprint

View full text Add to dashboard Cite

Speaker recognition is increasingly used in several everyday applications including smart speakers, customer care centers and other speech-driven analytics. It is crucial to accurately evaluate and mitigate biases present in machine learning (ML) based speech technologies, such as speaker recognition, to ensure their inclusive adoption. ML fairness studies with respect to various demographic factors in modern speaker recognition systems are lagging compared to other human-centered applications such as face recognition. Existing studies on fairness in speaker recognition systems are largely limited to evaluating biases at specific operating points of the systems, which can lead to false expectations of fairness. Moreover, there are only a handful of bias mitigation strategies developed for speaker recognition systems. In this paper, we systematically evaluate the biases present in speaker recognition systems with respect to gender across a range of system operating points. We also propose adversarial and multi-task learning techniques to improve the fairness of these systems. We show through quantitative and qualitative evaluations that the proposed methods improve the fairness of ASV systems over baseline methods trained using data balancing techniques. We also present a fairness-utility trade-off analysis to jointly examine fairness and the overall system performance. We show that although systems trained using adversarial techniques improve fairness, they are prone to reduced utility. On the other hand, multi-task methods can improve the fairness while retaining the utility. These findings can inform the choice of bias mitigation strategies in the field of speaker recognition.

show abstract

“…Data balancing [7] [23], [24] [13], [25], [26] AT [27], [28], [29] [14], [30], [31], [32], [33], [34] MTL [8] [35], [36], [37], [38]…”

Section: Asv Asr Other ML Domainsmentioning

confidence: 99%

Section: Fairness In Asvmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Peri¹,

Somandepalli²,

Narayanan³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Fenu et. al propose a benchmark to evaluate the fairness of end-to-end deep learning models with Thin-ResNet and X-vector architectures [4]. The study trains several models of young and old, female and male speakers in English and Spanish using the Mozilla Common Voice dataset.…”

Section: Fair Speaker Verificationmentioning

confidence: 99%

SVEva Fair: A Framework for Evaluating Fairness in Speaker Verification

Toussaint¹,

Ding²

2021

Preprint

View full text Add to dashboard Cite

Despite the success of deep neural networks (DNNs) in enabling on-device voice assistants, increasing evidence of bias and discrimination in machine learning is raising the urgency of investigating the fairness of these systems. Speaker verification is a form of biometric identification that gives access to voice assistants. Due to a lack of fairness metrics and evaluation frameworks that are appropriate for testing the fairness of speaker verification components, little is known about how model performance varies across subgroups, and what factors influence performance variation. To tackle this emerging challenge, we design and develop SVEva 1 Fair, an accessible, actionable and model-agnostic framework for evaluating the fairness of speaker verification components. The framework provides evaluation measures and visualisations to interrogate model performance across speaker subgroups and compare fairness between models. We demonstrate SVEva Fair in a case study with end-to-end DNNs trained on the VoxCeleb datasets to reveal potential bias in existing embedded speech recognition systems based on the demographic attributes of speakers. Our evaluation shows that publicly accessible benchmark models are not fair and consistently produce worse predictions for some nationalities, and for female speakers of most nationalities. To pave the way for fair and reliable embedded speaker verification, SVEva Fair has been implemented as an open-source python library and can be integrated into the embedded ML development pipeline to facilitate developers and researchers in troubleshooting unreliable speaker verification performance, and selecting high impact approaches for mitigating fairness challenges.

show abstract