Approaching human listener accuracy with modern speaker verification

Hautamäki, Ville; Kinnunen, Tomi; Nosratighods, Mohaddeseh; Lee, Kong Aik; Ma, Bin; Li, Haizhou

doi:10.21437/interspeech.2010-152

Cited by 18 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although machines outperform humans on long utterances in certain conditions (e.g., Hautam€ aki et al, 2010;Kahn et al, 2011), their performance on short utterances is seemingly worse than that of humans. For example, a state-ofthe-art text-independent ASV system using MFCCs was 97.60% accurate at discriminating speakers with 2.5-minlong pairs, but it was only 89.48% accurate with 5-s-long pairs, and performance worsened to 77.69% accuracy with 2-s-long pairs on the National Institute of Standards and Technology (NIST) speaker recognition evaluation (SRE) 2003 database , compared to 82.36% for humans hearing short utterances, as noted earlier.…”

Section: B Machine Speaker Verification and How It Compares To Human ...mentioning

confidence: 98%

Towards understanding speaker discrimination abilities in humans and machines for text-independent short utterances of different speech styles

Park

Yeung

Vesselinova

et al. 2018

The Journal of the Acoustical Society of America

View full text Add to dashboard Cite

Little is known about human and machine speaker discrimination ability when utterances are very short and the speaking style is variable. This study compares text-independent speaker discrimination ability of humans and machines based on utterances shorter than 2 s in two different speaking styles (read sentences and speech directed towards pets, characterized by exaggerated prosody). Recordings of 50 female speakers drawn from the UCLA Speaker Variability Database were used as stimuli. Performance of 65 human listeners was compared to i-vector-based automatic speaker verification systems using mel-frequency cepstral coefficients, voice quality features, which were inspired by a psychoacoustic model of voice perception, or their combination by score-level fusion. Humans always outperformed machines, except in the case of style-mismatched pairs from perceptually-marked speakers. Speaker representations by humans and machines were compared using multi-dimensional scaling (MDS). Canonical correlation analysis showed a weak correlation between machine and human MDS spaces. Multiple regression showed that means of voice quality features could represent the most important human MDS dimension well, but not the dimensions from machines. These results suggest that speaker representations by humans and machines are different, and machine performance might be improved by better understanding how different acoustic features relate to perceived speaker identity.

show abstract

Section: B Machine Speaker Verification and How It Compares To Human ...mentioning

confidence: 98%

Towards understanding speaker discrimination abilities in humans and machines for text-independent short utterances of different speech styles

Park

Yeung

Vesselinova

et al. 2018

The Journal of the Acoustical Society of America

View full text Add to dashboard Cite

show abstract

“…For example, to compute the FAR and FRR for the female population, we aggregated the verification trials where both enrolment and test utterance belonging to the female gender. Following Pereira and Mercel [40], we do not consider cross-gender trials (where enrolment and test utterances belong to different genders), because they tend to produce substantially lower FARs than same-gender trials [105]. To compute the demographic-agnostic FAR values useful for evaluating the auFaDR-FAR metric described in Section 4, we pooled all the verification trials agnostic to their demographic attributes.…”

Section: Evaluation Setupmentioning

confidence: 99%

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Peri¹,

Somandepalli²,

Narayanan³

2022

Preprint

View full text Add to dashboard Cite

Speaker recognition is increasingly used in several everyday applications including smart speakers, customer care centers and other speech-driven analytics. It is crucial to accurately evaluate and mitigate biases present in machine learning (ML) based speech technologies, such as speaker recognition, to ensure their inclusive adoption. ML fairness studies with respect to various demographic factors in modern speaker recognition systems are lagging compared to other human-centered applications such as face recognition. Existing studies on fairness in speaker recognition systems are largely limited to evaluating biases at specific operating points of the systems, which can lead to false expectations of fairness. Moreover, there are only a handful of bias mitigation strategies developed for speaker recognition systems. In this paper, we systematically evaluate the biases present in speaker recognition systems with respect to gender across a range of system operating points. We also propose adversarial and multi-task learning techniques to improve the fairness of these systems. We show through quantitative and qualitative evaluations that the proposed methods improve the fairness of ASV systems over baseline methods trained using data balancing techniques. We also present a fairness-utility trade-off analysis to jointly examine fairness and the overall system performance. We show that although systems trained using adversarial techniques improve fairness, they are prone to reduced utility. On the other hand, multi-task methods can improve the fairness while retaining the utility. These findings can inform the choice of bias mitigation strategies in the field of speaker recognition.

show abstract

“…As a way of modelling the perceptual experiences of listeners with voices, ASV systems typically train models on large datasets with many different speakers. Only a handful of studies have compared human and machine performances, which have focused on the effects of speech types, such as the length of utterances [17,22,48]. However, to the authors' knowledge no findings have been reported with regards to the effectiveness of automatic scores to model listener responses across perceptual tasks.…”

Section: Introductionmentioning

confidence: 99%

Evaluating the effects of task design on unfamiliar Francophone listener and automatic speaker identification performance

O’Brien

Meunier

Tomashenko

et al. 2023

Multimed Tools Appl

View full text Add to dashboard Cite

Many questions remain with regards to how context affects perceptual and automatic speaker identification performance. To examine the effects of task design on perceptual speaker identification performance, three tasks were developed, including lineup and binary tasks, as well as a novel clustering task. Speech recordings of native French speakers were compared similarly across tasks evaluated by unfamiliar Francophone listeners. True positive (sensitivity) and true negative (specificity) response rates across tasks were measured. Our results showed participants had similar sensitivity and specificity for binary (88%) and clustering (84%) tasks, but random selection rates for the lineup task. Pearson correlation procedures were used to evaluate the efficiency of scores produced by a state-of-the-art automatic speaker verification to model perceptual responses (equal error rate = 89%). Automatic scores modelled lineup (r 2 = 0.6) and clustering (r 2 = 0.5) task accuracy quite well, however, they were less robust when modelling binary task responses (r 2 = -0.2). The results underscore the role task design plays in shaping perceptual responses, which, in turn, affects the modelling effectiveness of automatic scores. As evidence points to humans and algorithms modelling speakers differently, our findings suggest automatic speaker identification performance might be improved with a greater understanding on how context shapes perceptual responses.

show abstract

Approaching human listener accuracy with modern speaker verification

Cited by 18 publications

References 8 publications

Towards understanding speaker discrimination abilities in humans and machines for text-independent short utterances of different speech styles

Towards understanding speaker discrimination abilities in humans and machines for text-independent short utterances of different speech styles

To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition

Evaluating the effects of task design on unfamiliar Francophone listener and automatic speaker identification performance

Contact Info

Product

Resources

About