Age-Invariant Training for End-to-End Child Speech Recognition Using Adversarial Multi-Task Learning

Rumberg, Lars; Ehlert, Hanna; Lüdtke, Ulrike M.; Östermann, Jörn

doi:10.21437/interspeech.2021-1241

Cited by 5 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, our previous work [16] found that the FHVAE does not separate the dysarthria and content information and speech impairment is identifiable from z i,n 1 . To obtain dysarthria-invariant features z i,n 2 , inspired by [17], we introduce adversarial training into the FHVAE model. The data flow of the proposed model is provided in figure 1.…”

Section: Fhvae With Adversarial Trainingmentioning

confidence: 99%

“…To achieve a greater deal of dysarthria-invariance in the content variable, instead of using forced regularization with phoneme alignment [16], inspired by [17], we introduce adversarial training [18] to the FHVAE model. We feed the FH-VAE model with both control and dysarthric speech, regarding the content variable encoder as the generator.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Weak-Supervised Dysarthria-Invariant Features for Spoken Language Understanding Using an Fhvae and Adversarial Training

hamme

2023

2022 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

The scarcity of training data and the large speaker variation in dysarthric speech lead to poor accuracy and poor speaker generalization of spoken language understanding systems for dysarthric speech. Through work on the speech features, we focus on improving the model generalization ability with limited dysarthric data. Factorized Hierarchical Variational Auto-Encoders (FHVAE) trained unsupervisedly have shown their advantage in disentangling content and speaker representations. Earlier work showed that the dysarthria shows in both feature vectors. Here, we add adversarial training to bridge the gap between the control and dysarthric speech data domains. We extract dysarthric and speaker invariant features using weak supervision. The extracted features are evaluated on a Spoken Language Understanding task and yield a higher accuracy on unseen speakers with more severe dysarthria compared to features from the basic FHVAE model or plain filterbanks.

show abstract

Section: Fhvae With Adversarial Trainingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Weak-Supervised Dysarthria-Invariant Features for Spoken Language Understanding Using an Fhvae and Adversarial Training

hamme

2023

2022 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

show abstract

“…Prior research from the Speech Technology community on Children ASR (Stemmer et al, 2003;Shivakumar et al, 2014;Tong et al, 2017;Wu et al, 2019;Shivakumar and Georgiou, 2020;Yeung et al, 2021;Rumberg et al, 2021;Gretter et al, 2021) is captivating. But these focused on: (i) older children, including kindergarten (6-15 yrs), (ii) data collected using head-mounted microphones or close-proximity handheld smartphones in clean/controlled settings under adult supervision, and (iii) with just one speaker using prompts or read stimuli, and limited spontaneous (not scripted) speech.…”

Section: Challenges Of Developing Automaticmentioning

confidence: 99%

Activity focused Speech Recognition of Preschool Children in Early Childhood Classrooms

Dutta¹,

Irvin²,

Buzhardt³

et al. 2022

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

View full text Add to dashboard Cite

A supportive environment is vital for overall cognitive development in children. Challenges with direct observation and limitations of access to data driven approaches often hinder teachers or practitioners in early childhood research to modify or enhance classroom structures. Deploying sensor based tools in naturalistic preschool classrooms will thereby help teachers/practitioners to make informed decisions and better support student learning needs. In this study, two elements of eco-behavioral assessment: conversational speech and realtime location are fused together. While various challenges remain in developing Automatic Speech Recognition systems for spontaneous preschool children speech, efforts are made to develop a hybrid ASR engine reporting an effective Word-Error-Rate of 40%. The ASR engine further supports recognition of spoken words, WH-words, and verbs in various activity learning zones in a naturalistic preschool classroom scenario. Activity areas represent various locations within the physical ecology of an early childhood setting, each of which is suited for knowledge and skill enhancement in young children. Capturing children's communication engagement in such areas could help teachers/practitioners fine-tune their daily activities, without the need for direct observation. This investigation provides evidence of the use of speech technology in educational settings to better support such early childhood intervention.

show abstract

“…An alternative to fine-tuning is multitask learning, where the model is trained on both domains simultaneously. Tong et al [33] explored multitask learning for child and adult speech recognition, and Rumberg et al [50] constrained the model to learn features that are independent of speaker age, leading to better transfer between the domains. Other work augmented adult speech by making it more similar to child speech; this involved simulating phenomena such as vowel prolongation that are typically associated with speech produced by children [51].…”

Section: Transcription (3)mentioning

confidence: 99%

Multidisciplinary Perspectives on Automatic Analysis of Children’s Language Samples: Where Do We Go from Here?

Lüdtke

Bornman

Wet

et al. 2022

Folia Phoniatr Logop

View full text Add to dashboard Cite

Background: Language sample analysis (LSA) is invaluable to describe and understand child language use and development for clinical purposes and research. Digital tools supporting LSA are available, but many of the LSA steps have not been automated. Nevertheless, programs that include automatic speech recognition (ASR), the first step of LSA, have already reached mainstream applicability. Summary: To better understand the complexity, challenges and future needs of automatic LSA, including the tasks of transcribing, annotating and analysing natural child language samples, this article takes on a multi-disciplinary view. Requirements of a fully automated LSA process are characterized, features of existing LSA software tools compared, and prior work from the disciplines of information science and computational linguistics reviewed. Key Messages: Existing tools vary in their extent of automation provided across the process of LSA. Advances in machine learning for speech recognition and processing have potential to facilitate LSA, but the specifics of child speech and language as well as the lack of child data complicate software design. A transdisciplinary approach is recommended as feasible to support future software development for LSA.

show abstract

Age-Invariant Training for End-to-End Child Speech Recognition Using Adversarial Multi-Task Learning

Cited by 5 publications

References 23 publications

Weak-Supervised Dysarthria-Invariant Features for Spoken Language Understanding Using an Fhvae and Adversarial Training

Weak-Supervised Dysarthria-Invariant Features for Spoken Language Understanding Using an Fhvae and Adversarial Training

Activity focused Speech Recognition of Preschool Children in Early Childhood Classrooms

Multidisciplinary Perspectives on Automatic Analysis of Children’s Language Samples: Where Do We Go from Here?

Contact Info

Product

Resources

About