SVEva Fair: A Framework for Evaluating Fairness in Speaker Verification

Toussaint, Wiebke; Ding, Aaron Yi

doi:10.48550/arxiv.2107.12049

Cited by 4 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• voxceleb-H: Following Toussaint et al [9], we performed evaluations on the voxceleb-H split. It is a subset of Voxceleb1 containing 1190 speakers, and 500K verification trials consisting of same gender and same nationality pairs.…”

Section: Discussionmentioning

confidence: 99%

“…As noted by Drozdowski et al [76], a majority of bias detection and mitigation works in biometrics focus on face recognition [25,36,54,56], and some in fingerprint matching [77,78]. Fairness in voice-based biometrics remains to be an under-explored field with only a handful of works [7][8][9]79].…”

Section: Fairness In Asvmentioning

confidence: 99%

“…However, the evaluation of fairness was again limited to disparity in EER between the demographic groups. Toussaint et al [9] proposed an evaluation framework for probing the fairness of ASV systems. Their evaluations focused on the minimum detection cost function, which again considers one particular operating point characterized by a threshold on the speaker verification scores.…”

Section: Evaluating Biases In Asv Systemsmentioning

confidence: 99%

“…However, the primary source for lack of fair performance in ASV systems is the biases present in the speaker verification scores [50]. In order to understand the biases present in the ASV systems, we perform a qualitative analysis of the verification scores similar to the work by Touissant et al [9]. In particular, we plot the kernel density estimate plots of the cosine similarity scores of the impostor verification pairs for the female and male populations in Figure 5, and those of the genuine pairs in Figure 6.…”

Section: Biases In Verification Scoresmentioning

confidence: 99%

“…While tremendous strides have been made in evaluating and improving fairness in applied ML fields [1], techniques for bias mitigation in ASV systems are limited [7,8]. Notably, most current bias evaluation frameworks for ASV are restricted to specific operating points of the systems [9,10], limiting their applicability to specific use-cases. In particular, differences in the equal error rate (EER) of an ASV system between different demographic population groups is commonly used as a proxy for the biases present in that system [8,10].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Peri¹,

Somandepalli²,

Narayanan³

2022

Preprint

View full text Add to dashboard Cite

Speaker recognition is increasingly used in several everyday applications including smart speakers, customer care centers and other speech-driven analytics. It is crucial to accurately evaluate and mitigate biases present in machine learning (ML) based speech technologies, such as speaker recognition, to ensure their inclusive adoption. ML fairness studies with respect to various demographic factors in modern speaker recognition systems are lagging compared to other human-centered applications such as face recognition. Existing studies on fairness in speaker recognition systems are largely limited to evaluating biases at specific operating points of the systems, which can lead to false expectations of fairness. Moreover, there are only a handful of bias mitigation strategies developed for speaker recognition systems. In this paper, we systematically evaluate the biases present in speaker recognition systems with respect to gender across a range of system operating points. We also propose adversarial and multi-task learning techniques to improve the fairness of these systems. We show through quantitative and qualitative evaluations that the proposed methods improve the fairness of ASV systems over baseline methods trained using data balancing techniques. We also present a fairness-utility trade-off analysis to jointly examine fairness and the overall system performance. We show that although systems trained using adversarial techniques improve fairness, they are prone to reduced utility. On the other hand, multi-task methods can improve the fairness while retaining the utility. These findings can inform the choice of bias mitigation strategies in the field of speaker recognition.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Fairness In Asvmentioning

confidence: 99%

Section: Evaluating Biases In Asv Systemsmentioning

confidence: 99%

Section: Biases In Verification Scoresmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Peri¹,

Somandepalli²,

Narayanan³

2022

Preprint

View full text Add to dashboard Cite

show abstract

A Deep Neural Network for Short-Segment Speaker Recognition

Hajavi¹,

Etemad²

2019

Interspeech 2019

View full text Add to dashboard Cite

Today's interactive devices such as smart-phone assistants and smart speakers often deal with short-duration speech segments. As a result, speaker recognition systems integrated into such devices will be much better suited with models capable of performing the recognition task with short-duration utterances. In this paper, a new deep neural network, UtterIdNet, capable of performing speaker recognition with short speech segments is proposed. Our proposed model utilizes a novel architecture that makes it suitable for short-segment speaker recognition through an efficiently increased use of information in short speech segments. UtterIdNet has been trained and tested on the VoxCeleb datasets, the latest benchmarks in speaker recognition. Evaluations for different segment durations show consistent and stable performance for short segments, with significant improvement over the previous models for segments of 2 seconds, 1 second, and especially sub-second durations (250 ms and 500 ms).

show abstract

Privacy-Preserving Voice Anti-Spoofing Using Secure Multi-Party Computation

Chouchane¹,

Brossier²,

Gamboa³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

This study investigates the impact of gender information on utility, privacy, and fairness in voice biometric systems, guided by the General Data Protection Regulation (GDPR) mandates, which underscore the need for minimizing the processing and storage of private and sensitive data, and ensuring fairness in automated decision-making systems. We adopt an approach that involves the fine-tuning of the wav2vec 2.0 model for speaker verification tasks, evaluating potential gender-related privacy vulnerabilities in the process. Gender influences during the finetuning process were employed to enhance fairness and privacy in order to emphasise or obscure gender information within the speakers' embeddings. Results from VoxCeleb datasets indicate our adversarial model increases privacy against uninformed attacks, yet slightly diminishes speaker verification performance compared to the non-adversarial model. However, the model's efficacy reduces against informed attacks. Analysis of system performance was conducted to identify potential gender biases, thus highlighting the need for further research to understand and improve the delicate interplay between utility, privacy, and equity in voice biometric systems.

show abstract

SVEva Fair: A Framework for Evaluating Fairness in Speaker Verification

Cited by 4 publications

References 14 publications

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

A Deep Neural Network for Short-Segment Speaker Recognition

Privacy-Preserving Voice Anti-Spoofing Using Secure Multi-Party Computation

Contact Info

Product

Resources

About