Exploiting Cross Domain Acoustic-to-Articulatory Inverted Features for Disordered Speech Recognition

Hu, Shujie; Liu, Shansong; Xie, Xurong; Hu, Shoukang; Cui, Mingyu; Liu, Xunying; Meng, Helen

doi:10.1109/icassp43922.2022.9746989

Cited by 9 publications

(4 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their experimental results on the UASpeech corpus demonstrated that the AVSR based on cross-domain visual feature generation outperformed baseline ASR and AVSR without this approach. Hu, et al [130] proposed a cross-domain acoustic-to-articulator inversion approach. Their model was pre-trained using parallel acousticarticulatory data from the 15-hour TORGO corpus.…”

Section: Trends Of Avsr Technologies For Dysarthric Speechmentioning

confidence: 99%

A survey of technologies for automatic Dysarthric speech recognition

Qian,

Xiao,

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Speakers with dysarthria often struggle to accurately pronounce words and effectively communicate with others. Automatic speech recognition (ASR) is a powerful tool for extracting the content from speakers with dysarthria. However, the narrow concept of ASR typically only covers technologies that process acoustic modality signals. In this paper, we broaden the scope of this concept that the generalized concept of ASR for dysarthric speech. Our survey discussed the systems encompassed acoustic modality processing, articulatory movements processing and audio-visual modality fusion processing in the application of recognizing dysarthric speech. Contrary to previous surveys on dysarthric speech recognition, we have conducted a systematic review of the advancements in this field. In particular, we introduced state-of-the-art technologies to supplement the survey of recent research during the era of multi-modality fusion in dysarthric speech recognition. Our survey found that audio-visual fusion technologies perform better than traditional ASR technologies in the task of dysarthric speech recognition. However, training audio-visual fusion models requires more computing resources, and the available data corpus for dysarthric speech is limited. Despite these challenges, state-of-the-art technologies show promising potential for further improving the accuracy of dysarthric speech recognition in the future.

show abstract

Section: Trends Of Avsr Technologies For Dysarthric Speechmentioning

confidence: 99%

A survey of technologies for automatic Dysarthric speech recognition

Qian,

Xiao,

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

show abstract

“…and a 1h evaluation set (1892 utt.). Further speed perturbation [24], [104] produces a 34.1h augmented training set (61813 utt. ).…”

Section: A Experiments On Dysarthric Speechmentioning

confidence: 99%

Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

Xie

Wang

et al. 2022

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

The application of data-intensive automatic speech recognition (ASR) technologies to dysarthric and elderly adult speech is confronted by their mismatch against healthy and nonaged voices, data scarcity and large speaker-level variability. To this end, this paper proposes two novel data-efficient methods to learn homogeneous dysarthric and elderly speaker-level features for rapid, on-the-fly test-time adaptation of DNN/TDNN and Conformer ASR models. These include: 1) speaker-level varianceregularized spectral basis embedding (VR-SBE) features that exploit a special regularization term to enforce homogeneity of speaker features in adaptation; and 2) feature-based learning hidden unit contributions (f-LHUC) transforms that are conditioned on VR-SBE features. Experiments are conducted on four tasks across two languages: the English UASpeech and TORGO dysarthric speech datasets, the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora. The proposed on-the-fly speaker adaptation techniques consistently outperform baseline iVector and xVector adaptation by statistically significant word or character error rate reductions up to 5.32% absolute (18.57% relative) and batch-mode LHUC speaker adaptation by 2.24% absolute (9.20% relative), while operating with real-time factors speeding up to 33.6 times against xVectors during adaptation. The efficacy of the proposed adaptation techniques is demonstrated in a comparison against current ASR technologies including SSL pre-trained systems on UASpeech, where our best system produces a state-of-the-art WER of 23.33%. Analyses show VR-SBE features and f-LHUC transforms are insensitive to speaker-level data quantity in testtime adaptation. T-SNE visualization reveals they have stronger speaker-level homogeneity than baseline iVectors, xVectors and batch-mode LHUC transforms.

show abstract

“…After removal of excessive silence, the training and test set contain 6.5 hours (14541 utterances) and 1 hour (1892 utterances) of speech respectively. After data augmentation with both speaker dependent and speaker independent speed perturbation [15], [126], the augmented training set contains 34.1 hours of data (61813 utterances).…”

Section: A Experiments On Dysarthric Speechmentioning

confidence: 99%

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Jin¹,

Deng²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. It is difficult to collect large quantities of such data for ASR system development due to the mobility issues often found among these users. To this end, data augmentation techniques play a vital role. In contrast to existing data augmentation techniques only modifying the speaking rate or overall shape of spectral contour, fine-grained spectro-temporal differences between dysarthric, elderly and normal speech are modelled using a novel set of speaker dependent (SD) generative adversarial networks (GAN) based data augmentation approaches in this paper. These flexibly allow both: a) temporal or speed perturbed normal speech spectra to be modified and closer to those of an impaired speaker when parallel speech data is available; and b) for non-parallel data, the SVD decomposed normal speech spectral basis features to be transformed into those of a target elderly speaker before being re-composed with the temporal bases to produce the augmented data for state-of-the-art TDNN and Conformer ASR system training. Experiments are conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed GAN based data augmentation approaches consistently outperform the baseline speed perturbation method by up to 0.91% and 3.0% absolute (9.61% and 6.4% relative) WER reduction on the TORGO and DementiaBank data respectively. Consistent performance improvements are retained after applying LHUC based speaker adaptation.

show abstract

Exploiting Cross Domain Acoustic-to-Articulatory Inverted Features for Disordered Speech Recognition

Cited by 9 publications

References 35 publications

A survey of technologies for automatic Dysarthric speech recognition

A survey of technologies for automatic Dysarthric speech recognition

Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Contact Info

Product

Resources

About