Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Abstract

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. It is difficult to collect large quantities of such data for ASR system development due to the mobility issues often found among these users. To this end, data augmentation techniques play a vital role. In contrast to existing data augmentation techniques only modifying the speaking rate or overall shape of spectral contour, fine-grained spectro-temporal differences between dysarthric, elderly and normal speech are modelled using a novel set of speaker dependent (SD) generative adversarial networks (GAN) based data augmentation approaches in this paper. These flexibly allow both: a) temporal or speed perturbed normal speech spectra to be modified and closer to those of an impaired speaker when parallel speech data is available; and b) for non-parallel data, the SVD decomposed normal speech spectral basis features to be transformed into those of a target elderly speaker before being re-composed with the temporal bases to produce the augmented data for state-of-the-art TDNN and Conformer ASR system training. Experiments are conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed GAN based data augmentation approaches consistently outperform the baseline speed perturbation method by up to 0.91% and 3.0% absolute (9.61% and 6.4% relative) WER reduction on the TORGO and DementiaBank data respectively. Consistent performance improvements are retained after applying LHUC based speaker adaptation.

Weak-Supervised Dysarthria-Invariant Features for Spoken Language Understanding Using an Fhvae and Adversarial Training

“…A second approach is to decrease the model size [7], or to train an inserted small module instead of finetuning the whole model [8,9], so the number of parameters learned on the dysarthric data is limited. Thirdly and differently from the solutions that work on training strategy or model structure, [10,11,12,13] focus directly on the data and do augmentation to generate more dysarthric speech for use in training.…”

Confidence Score Based Conformer Speaker Adaptation for Speech Recognition

“…For example, dysarthric speakers of very low speech intelligibility exhibit clearer patterns of articulatory imprecision, decreased volume and clarity, increased dysfluencies, slower speaking rate and changes in pitch [29], while those diagonalized with mid or high speech intelligibility are closer to normal speakers. Such heterogeneity further increases the mismatch against normal speech and the difficulty in both speaker-independent (SI) ASR system development using limited impaired speech data and fine-grained personalization to individual users' data [3,25,30] So far the majority of prior researches to address the dysarthric speaker level diversity have been focused on using speaker-identity only either in speaker-dependent (SD) data augmentation [7,9,13,14,18,27], or in speaker adapted or dependent ASR system development [1, 3, 4, 7, 11-13, 19, 22, 25, 31-33]. In contrast, very limited prior researches have used speech impairment severity information for dysarthric speech recognition.…”

“…A set of novel techniques and recipe configurations were proposed to learn both speech impairment severity and speaker-identity when constructing and personalizing these systems. In contrast, prior researches mainly focused on using speaker-identity only in speaker-dependent data augmentation [7,9,13,14,18,27] and speaker adapted or dependent ASR system development [1,3,4,7,11,13,19,22,23,25,[31][32][33]. Very limited prior researches utilized speech impairment severity information [2,11,25,[34][35][36].…”