Speaker adaptation for Wav2vec2 based dysarthric ASR

Baskar, Murali Karthick; Herzig, Tim; Nguyen, Diana; Díez, Mireia; Polzehl, Tim; Burget, Lukáš; Černocký, Jaň

doi:10.21437/interspeech.2022-10896

Cited by 19 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, limited previous speaker adaptation research has been conducted for E2E ASR systems. Among these, auxiliary speaker adaptive features based on i-vector [21], [23], x-vector [23], [60] and f-MLLR [60], or extracted from sequence summary network [61], attention-based speaker memory [62] and speaker-aware modules [63], [64] are incorporated into attention-based encoder-decoder, or conventional, non-convolution augmented Transformer models. Several model-based speaker adaptation methods for E2E models utilize speaker adaptive neural network internal components, for example, SD neural beamforming, encoder, attention or decoder modules in multichannel E2E systems [65], while keeping the parameters of other components fixed as speaker independent during adaptation.…”

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

“…In contrast, prior research works on structured transform-based speaker adaptation were limited to the conventional hybrid DNN-HMM based ASR systems [36]- [49]. In contrast, existing speaker adaptation methods designed for E2E ASR systems largely focused on using auxiliary speaker adaptive features [21], [23], [60]- [64], or directly adapting the whole or certain components of the SI models [22], [65]- [67]. The speaker-level data sparsity issue encountered in these dataintensive model-based adaptation approaches remains largely unaddressed.…”

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

“…The speaker-level data sparsity issue encountered in these dataintensive model-based adaptation approaches remains largely unaddressed. Prior researches also investigated the use of auxiliary speaker embedding-based approaches for E2E systems [21], [23], [60], [61]. However, these were found to produce inconsistent results in speaker adaptation, e.g., for the IBM Conformer systems reported on the Switchboard task [21].…”

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

See 2 more Smart Citations

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Xie

Wang

Cui

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Speaker adaptation techniques provide a powerful solution to customise automatic speech recognition (ASR) systems for individual users. Practical application of unsupervised modelbased speaker adaptation techniques to data intensive end-to-end ASR systems is hindered by the scarcity of speaker-level data and performance sensitivity to transcription errors. To address these issues, a set of compact and data efficient speaker-dependent (SD) parameter representations are used to facilitate both speaker adaptive training and test-time unsupervised speaker adaptation of state-of-the-art Conformer ASR systems. The sensitivity to supervision quality is reduced using a confidence score-based selection of the less erroneous subset of speaker-level adaptation data. Two lightweight confidence score estimation modules are proposed to produce more reliable confidence scores. The data sparsity issue, which is exacerbated by data selection, is addressed by modelling the SD parameter uncertainty using Bayesian learning. Experiments on the benchmark 300-hour Switchboard and the 233-hour AMI datasets suggest that the proposed confidence score-based adaptation schemes consistently outperformed the baseline speaker-independent (SI) Conformer model and conventional non-Bayesian, point estimate-based adaptation using no speaker data selection. Similar consistent performance improvements were retained after external Transformer and LSTM language model rescoring. In particular, on the 300-hour Switchboard corpus, statistically significant WER reductions of 1.0%, 1.3%, and 1.4% absolute (9.5%, 10.9%, and 11.3% relative) were obtained over the baseline SI Conformer on the NIST Hub5'00, RT02, and RT03 evaluation sets respectively. Similar WER reductions of 2.7% and 3.3% absolute (8.9% and 10.2% relative) were also obtained on the AMI development and evaluation sets.

show abstract

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Xie

Wang

Cui

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…Recent efforts on dysarthric speech recognition focused on personalized or tuned ASR models (e.g., [4,5,6,7,8,9,10]), which leverage large proprietary or non-commercial datasets of atypical speech (e.g., Project Euphonia [1]; UASpeech [11]; AphasiaBank [12]). In this work, we take a more pragmatic approach that does not require vast quantities of data, yet enables people with severe speech differences to train phrase recognition models for applications where only a constrained set of phrases is needed.…”

Section: Introductionmentioning

confidence: 99%

Nonverbal Sound Detection for Disordered Speech

Huang

Jain

Tooley

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Many consumer speech recognition systems are not tuned for people with speech disabilities, resulting in poor recognition and user experience, especially for severe speech differences. Recent studies have emphasized interest in personalized speech models from people with atypical speech patterns. We propose a query-by-example-based personalized phrase recognition system that is trained using small amounts of speech, is language agnostic, does not assume a traditional pronunciation lexicon, and generalizes well across speech difference severities. On an internal dataset collected from 32 people with dysarthria, this approach works regardless of severity and shows a 60% improvement in recall relative to a commercial speech recognition system. On the public EasyCall dataset of dysarthric speech, our approach improves accuracy by 30.5%. Performance degrades as the number of phrases increases, but consistently outperforms ASR systems when trained with 50 unique phrases.

show abstract

“…A set of novel techniques and recipe configurations were proposed to learn both speech impairment severity and speaker-identity when constructing and personalizing these systems. In contrast, prior researches mainly focused on using speaker-identity only in speaker-dependent data augmentation [7,9,13,14,18,27] and speaker adapted or dependent ASR system development [1,3,4,7,11,13,19,22,23,25,[31][32][33]. Very limited prior researches utilized speech impairment severity information [2,11,25,[34][35][36].…”

Section: Introductionmentioning

confidence: 99%

Confidence Score Based Conformer Speaker Adaptation for Speech Recognition

Xie¹,

Wang²,

Cui³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

A key challenge in dysarthric speech recognition is the speaker-level diversity attributed to both speaker-identity associated factors such as gender, and speech impairment severity. Most prior researches on addressing this issue focused on using speaker-identity only. To this end, this paper proposes a novel set of techniques to use both severity and speaker-identity in dysarthric speech recognition: a) multitask training incorporating severity prediction error; b) speaker-severity aware auxiliary feature adaptation; and c) structured LHUC transforms separately conditioned on speaker-identity and severity. Experiments conducted on UASpeech suggest incorporating additional speech impairment severity into state-of-the-art hybrid DNN, E2E Conformer and pre-trained Wav2vec 2.0 ASR systems produced statistically significant WER reductions up to 4.78% (14.03% relative). Using the best system the lowest published WER of 17.82% (51.25% on very low intelligibility) was obtained on UASpeech.

show abstract

Speaker adaptation for Wav2vec2 based dysarthric ASR

Cited by 19 publications

References 0 publications

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Nonverbal Sound Detection for Disordered Speech

Confidence Score Based Conformer Speaker Adaptation for Speech Recognition

Contact Info

Product

Resources

About