A Comparative Study of Speech Anonymization Metrics

Maouche, Mohamed; Srivastava, Brij Mohan Lal; Vauquier, Nathalie; Bellet, Aurélien; Tommasi, Marc

doi:10.21437/interspeech.2020-2248

Cited by 13 publications

(17 citation statements)

References 22 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such guidelines should determine acceptable FPR and FNR ranges, and guide normative decisions pertaining to the selection of weights of cost functions. Alternative evaluation metrics, such as those used for privacy-preserving speaker verification [24,35], should also be studied for evaluating bias. Lastly, to assess aggregation bias in speaker verification, disaggreated evaluation across speaker subgroups is needed.…”

Section: Recommendationsmentioning

confidence: 99%

Bias in Automated Speaker Recognition

Hutiri,

Ding

2022

Preprint

View full text Add to dashboard Cite

Automated speaker recognition uses data processing to identify speakers by their voice. Today, automated speaker recognition technologies are deployed on billions of smart devices and in services such as call centres. Despite their wide-scale deployment and known sources of bias in face recognition and natural language processing, bias in automated speaker recognition has not been studied systematically. We present an in-depth empirical and analytical study of bias in the machine learning development workflow of speaker verification, a voice biometric and core task in automated speaker recognition. Drawing on an established framework for understanding sources of harm in machine learning, we show that bias exists at every development stage in the well-known VoxCeleb Speaker Recognition Challenge, including model building, implementation, and data generation. Most affected are female speakers and non-US nationalities, who experience significant performance degradation. Leveraging the insights from our findings, we make practical recommendations for mitigating bias in automated speaker recognition, and outline future research directions.

show abstract

Section: Recommendationsmentioning

confidence: 99%

Bias in Automated Speaker Recognition

Hutiri,

Ding

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To obtain the linkability score for a given speaker, the average is taken from all mated scores of this specific speaker. Work in [6] advocate the use of D sys ↔ as a robust privacy metric. The lower the D sys ↔ , the better the speakers are anonymized.…”

Section: Utility and Privacy Metricsmentioning

confidence: 99%

Evaluating X-Vector-Based Speaker Anonymization Under White-Box Assessment

Champion

Jouvet

Larcher

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In the scenario of the Voice Privacy challenge, anonymization is achieved by converting all utterances from a source speaker to match the same target identity; this identity being randomly selected. In this context, an attacker with maximum knowledge about the anonymization system can not infer the target identity. This article proposed to constrain the target selection to a specific identity, i.e., removing the random selection of identity, to evaluate the extreme threat under a whitebox assessment (the attacker has complete knowledge about the system). Targeting a unique identity also allows us to investigate whether some target's identities are better than others to anonymize a given speaker.

show abstract

“…The ongoing work on privacy preservation assessment is focusing on the development of new eval-uation frameworks, anonymization metrics, and investigation of their correlation and complementarity. This includes the ZEBRA framework (Nautsch et al, 2020;Noé et al, 2021), objective and subjective linkability metrics (Maouche et al, 2020). Also one may be interested in evaluation that is close to real industry applications and tasks, for example, speaker labeling for diarization, analysis of time and quality required for annotation of real vs anonymized speech (Espinoza-Cuadros et al, 2020b).…”

Section: Open Questions and Future Directionsmentioning

confidence: 99%

The VoicePrivacy 2020 Challenge: Results and findings

Tomashenko,

Wang,

Vincent

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents the results and analyses stemming from the first VoicePrivacy 2020 Challenge which focuses on developing anonymization solutions for speech technology. We provide a systematic overview of the challenge design with an analysis of submitted systems and evaluation results. In particular, we describe the voice anonymization task and datasets used for system development and evaluation. Also, we present different attack models and the associated objective and subjective evaluation metrics. We introduce two anonymization baselines and provide a summary description of the anonymization systems developed by the challenge participants. We report objective and subjective evaluation results for baseline and submitted systems. In addition, we present experimental results for alternative privacy metrics and attack models developed as a part of the post-evaluation analysis. Finally, we summarise our insights and observations that will influence the design of the next VoicePrivacy challenge edition and some directions for future voice anonymization research.

show abstract

A Comparative Study of Speech Anonymization Metrics

Cited by 13 publications

References 22 publications

Bias in Automated Speaker Recognition

Bias in Automated Speaker Recognition

Evaluating X-Vector-Based Speaker Anonymization Under White-Box Assessment

The VoicePrivacy 2020 Challenge: Results and findings

Contact Info

Product

Resources

About