Adversarial Data Augmentation for Disordered Speech Recognition

Abstract

Automatic recognition of disordered speech remains a highly challenging task to date. The underlying neuro-motor conditions, often compounded with co-occurring physical disabilities, lead to the difficulty in collecting large quantities of impaired speech required for ASR system development. This paper presents novel variational auto-encoder generative adversarial network (VAE-GAN) based personalized disordered speech augmentation approaches that simultaneously learn to encode, generate and discriminate synthesized impaired speech. Separate latent features are derived to learn dysarthric speech characteristics and phoneme context representations. Selfsupervised pre-trained Wav2vec 2.0 embedding features are also incorporated. Experiments conducted on the UASpeech corpus suggest the proposed adversarial data augmentation approach consistently outperformed the baseline speed perturbation and non-VAE GAN augmentation methods with trained hybrid TDNN and End-toend Conformer systems. After LHUC speaker adaptation, the best system using VAE-GAN based augmentation produced an overall WER of 27.78% on the UASpeech test set of 16 dysarthric speakers, and the lowest published WER of 57.31% on the subset of speakers with "Very Low" intelligibility.

Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition

“…In contrast, related previous works either used: a) synthesized normal speech acoustic-articulatory features trained A2A inversion models before being applied to dysarthric speech [23], while the large mismatch between normal and impaired speech encountered during inversion model training and articulatory feature generation stages was not taken into account; or b) only considered the cross-domain or cross-corpus A2A inversion [25] while the quality of generated articulatory features was not assessed using the back-end disordered speech recognition systems. In addition, the lowest published WER of 24.82% on the benchmark UASpeech task in comparison against recent researches [8][9][10][11][12][13][37][38][39] was obtained using the proposed cross-domain acoustic-to-articulatory inversion approach.…”

“…Data augmentation techniques play a vital role to address the data sparsity problem in current disordered speech recognition systems [37,38]. Spectral-temporal perturbation of the limited audio data collected from impaired speakers is normally used to inject more diversity into the augmented data to improve the resulting ASR system generalization on the same task, for example, the TORGO corpus.…”

“…"DA" stands for data augmentation. systems WER% Sheffield-2013 Cross domain augmentation [8] 37.50 Sheffield-2015 Speaker adaptive training [9] 34.80 CUHK-2018 DNN System Combination [10] 30.60 Sheffield-2020 Fine-tuning CNN-TDNN speaker adaptation [11] 30.76 CUHK-2020 Cross-domain AVSR [12] 26.84 CUHK-2020 DNN + DA + LHUC SAT [37] 26.37 CUHK-2021 DNN + GAN based DA + LHUC SAT [38] 25.89 CUHK-2021 NAS DNN + DA + LHUC SAT + AV fusion [39] 25.21 DNN + DA + LHUC SAT + AAV fusion (ours) 24.82…”

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

“…In contrast, existing data augmentation studies for dysarthric and elderly speech mainly focus on using signal-level tempo or speed perturbation based methods [14], [15], [33], [69], [70]. The only previous research on GAN based dysarthric speech augmentation required the explicit use of parallel speech data of the UASpeech corpus [109], [110]. The non-parallel GAN based normal to pathological voice conversion approach studied in [111] is evaluated on naturalness and severity, but not measured in terms of the performance of ASR systems constructed using the generated data.…”

“…The overall architecture configurations of the proposed DC-GAN model follow our previous work [110], also again shown A flattening operation is applied to concatenate the outputs of convolutional layers, resulting in a 3000-dimensional vector.…”