Exploiting Cross-Domain Visual Feature Generation for Disordered Speech Recognition

Recent Progress in the CUHK Dysarthric Speech Recognition System

“…By incorporating all the above techniques, the best recognition system produced an overall word error rate (WER) of 25.21% on the 22.6-hour UASpeech test set containing 16 dysarthric speakers. To the best of our knowledge, this is the lowest WER published so far on the same task reported in the literature [8]- [10], [40], [45], [47]. An overall WER reduction of 5.39% absolute (17.61% relative) was obtained over the CUHK 2018 system featuring a 6-way DNN system combination [10] which defined state-of-the-art performance at the time.…”

“…Lastly, inspired by the bi-modal nature of human speech perception and the success of audio-visual speech recognition (AVSR) technologies when being applied to normal speech [42]- [44], visual information is further incorporated to improve disordered speech recognition performance. In order to address the data sparsity that arises from the difficulty to record large amounts of high quality audio-visual (AV) data, a cross-domain visual feature generation approach [45] was developed. High quality AV parallel data based on normal speech recording of the lip reading sentence (LRS2) dataset [46] was used to build neural AV inversion systems.…”

“…However, the resulting AV inversion system cannot be directly applied to dysarthric speech given its large mismatch against the normal speech data in the LRS2 corpus. This mismatch may render the generated visual features unreliable to use for subsequent AVSR system development [45], [83]. Such mismatch can be compensated using, for example, domain-adversarial neural network (DANN) [84] or multi-level adaptive network (MLAN) [64].…”

“…The lowest WER of 25.21% was obtained using the NAS auto-configured DNN AVSR system (Sys. [8] 37.50 CUHK-2021 LAS + CTC + Meta-learning + SAT [24] 35.00 Sheffield-2015 Speaker adaptive training [9] 34.80 Sheffield-2020 Fine-tuning CNN-TDNN speaker adaptation [85] 30.76 CUHK-2018 DNN system combination [10] 30.60 CUHK-2021 QuartzNet + CTC + Meta-learning + SAT [24] 30.50 Sheffield-2019 Kaldi TDNN + Data Aug. [47] 27.88 CUHK-2020 Cross-domain AVSR [45] 26.84 CUHK-2020 DNN + Data Aug. + LHUC SAT [40] 26.37 NAS DNN + Data Aug. + LHUC SAT + AV fusion (ours) 25.21…”

Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition

“…In spite of the swift progress of automatic speech recognition (ASR) technologies targeting normal speech in the past few decades [1][2][3][4][5][6][7][8][9], accurate recognition of disordered speech remains a demanding task to date [10][11][12][13][14][15][16]. The underlying causes of speech disorders include a wide range of neuro-motor conditions, such as cerebral palsy, Parkinson disease, amyotrophic lateral sclerosis and stroke or traumatic brain injuries [17].…”