Weak-Supervised Dysarthria-Invariant Features for Spoken Language Understanding Using an Fhvae and Adversarial Training

Abstract

The scarcity of training data and the large speaker variation in dysarthric speech lead to poor accuracy and poor speaker generalization of spoken language understanding systems for dysarthric speech. Through work on the speech features, we focus on improving the model generalization ability with limited dysarthric data. Factorized Hierarchical Variational Auto-Encoders (FHVAE) trained unsupervisedly have shown their advantage in disentangling content and speaker representations. Earlier work showed that the dysarthria shows in both feature vectors. Here, we add adversarial training to bridge the gap between the control and dysarthric speech data domains. We extract dysarthric and speaker invariant features using weak supervision. The extracted features are evaluated on a Spoken Language Understanding task and yield a higher accuracy on unseen speakers with more severe dysarthria compared to features from the basic FHVAE model or plain filterbanks.

Models and Approaches for Comprehension of Dysarthric Speech Using Natural Language Processing: Systematic Review

“…Studies that emphasized speech patterns [23,27,[36][37][38][39][40][41][42] were interested in the formation of words spoken, omissions in their patterns, and inclusion of interesting vocabulary during discourse. These studies were much aligned toward word representation and drawing meaning out of the same by leveraging other speech-independent features, such as the speakers' emotions.…”

“…The studies that used vector encoding [15,[38][39][40][41][42][43]50,51] used NLU-based models, such long short-term memory neural networks or combinations of gated recurrent unit and convolutional neural networks to achieve the tasks of dialogue assessment in dysarthric speech, language understanding, and semantic pattern tracking.…”

“…Extensive and well-documented databases, such as the TORGO database and the UA-Speech database, contain data sets that have the potential to yield the in-depth speech patterns necessary for speech comprehension. Studies that used these 2 databases [26,31] (n=2; 6% of the studies) or a hybrid of any other databases (n=11; 37% of the studies) [28,33,[37][38][39][40][41][47][48][49]51] focused more on the application of the data in their proposed models, with little effort going into curation and preprocessing of the data.…”