Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

Abstract

The application of data-intensive automatic speech recognition (ASR) technologies to dysarthric and elderly adult speech is confronted by their mismatch against healthy and nonaged voices, data scarcity and large speaker-level variability. To this end, this paper proposes two novel data-efficient methods to learn homogeneous dysarthric and elderly speaker-level features for rapid, on-the-fly test-time adaptation of DNN/TDNN and Conformer ASR models. These include: 1) speaker-level varianceregularized spectral basis embedding (VR-SBE) features that exploit a special regularization term to enforce homogeneity of speaker features in adaptation; and 2) feature-based learning hidden unit contributions (f-LHUC) transforms that are conditioned on VR-SBE features. Experiments are conducted on four tasks across two languages: the English UASpeech and TORGO dysarthric speech datasets, the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora. The proposed on-the-fly speaker adaptation techniques consistently outperform baseline iVector and xVector adaptation by statistically significant word or character error rate reductions up to 5.32% absolute (18.57% relative) and batch-mode LHUC speaker adaptation by 2.24% absolute (9.20% relative), while operating with real-time factors speeding up to 33.6 times against xVectors during adaptation. The efficacy of the proposed adaptation techniques is demonstrated in a comparison against current ASR technologies including SSL pre-trained systems on UASpeech, where our best system produces a state-of-the-art WER of 23.33%. Analyses show VR-SBE features and f-LHUC transforms are insensitive to speaker-level data quantity in testtime adaptation. T-SNE visualization reveals they have stronger speaker-level homogeneity than baseline iVectors, xVectors and batch-mode LHUC transforms.

Confidence Score Based Conformer Speaker Adaptation for Speech Recognition

“…The underlying variability of dysarthric speech manifesting in changes of spectral envelope, volume reduction, imprecise articulation and breathy or horse voices, can be modeled via disentangling the speech spectrum into time-invariant and time-variant subspaces [11,30] learned in a supervised manner [11,25]. The resulting spectral basis deep embedding (SBE) features are more effective in encoding latent attributes of im-paired speech [25] than classical speaker embeddings such as iVectors [44] and xVectors [45]. Hence, we adopt SBE features as speaker-severity aware auxiliary inputs to the hybrid DNN systems to incorporate both speaker-identity and speech impairment severity into the acoustic front-ends.…”

“…For example, dysarthric speakers of very low speech intelligibility exhibit clearer patterns of articulatory imprecision, decreased volume and clarity, increased dysfluencies, slower speaking rate and changes in pitch [29], while those diagonalized with mid or high speech intelligibility are closer to normal speakers. Such heterogeneity further increases the mismatch against normal speech and the difficulty in both speaker-independent (SI) ASR system development using limited impaired speech data and fine-grained personalization to individual users' data [3,25,30] So far the majority of prior researches to address the dysarthric speaker level diversity have been focused on using speaker-identity only either in speaker-dependent (SD) data augmentation [7,9,13,14,18,27], or in speaker adapted or dependent ASR system development [1, 3, 4, 7, 11-13, 19, 22, 25, 31-33]. In contrast, very limited prior researches have used speech impairment severity information for dysarthric speech recognition.…”

“…To this end, this paper investigates a novel set of techniques to incorporate speech impairment severity into state-ofthe-art hybrid DNN [13], end-to-end Conformer [37] and selfsupervised learning (SSL) based pre-trained Wav2vec 2.0 [38] ASR systems. These include the use of: a) multi-task [39] training cost interpolation between the ASR loss and speech impairment severity prediction error; b) spectral basis embedding (SBE) [11,25] based speaker-severity aware adaptation features that are trained to simultaneously predict both speaker-identity and impairment severity; and c) structured learning hidden units contribution (LHUC) [40] transforms that are separately conditioned on speaker-identity and impairment severity. These are used to facilitate both speaker-severity adaptive training of ASR systems and their test-time unsupervised adaptation to both factors of variability.…”

“…A set of novel techniques and recipe configurations were proposed to learn both speech impairment severity and speaker-identity when constructing and personalizing these systems. In contrast, prior researches mainly focused on using speaker-identity only in speaker-dependent data augmentation [7,9,13,14,18,27] and speaker adapted or dependent ASR system development [1,3,4,7,11,13,19,22,23,25,[31][32][33]. Very limited prior researches utilized speech impairment severity information [2,11,25,[34][35][36].…”

“…In contrast, prior researches mainly focused on using speaker-identity only in speaker-dependent data augmentation [7,9,13,14,18,27] and speaker adapted or dependent ASR system development [1,3,4,7,11,13,19,22,23,25,[31][32][33]. Very limited prior researches utilized speech impairment severity information [2,11,25,[34][35][36]. None of these was conducted in the context of fine-tuning state-of-the-art pre-trained ASR models for dysarthric speech recognition, as considered in this paper.…”