Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition

Liu, Shansong; Yu, Jianwei; Xie, Xurong; Hu, Shoukang; Ye, Zi; Jin, Zengrui; Liu, Xunying; Meng, Helen

doi:10.21437/interspeech.2021-60

Cited by 11 publications

(10 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, related previous works either used: a) synthesized normal speech acoustic-articulatory features trained A2A inversion models before being applied to dysarthric speech [23], while the large mismatch between normal and impaired speech encountered during inversion model training and articulatory feature generation stages was not taken into account; or b) only considered the cross-domain or cross-corpus A2A inversion [25] while the quality of generated articulatory features was not assessed using the back-end disordered speech recognition systems. In addition, the lowest published WER of 24.82% on the benchmark UASpeech task in comparison against recent researches [8][9][10][11][12][13][37][38][39] was obtained using the proposed cross-domain acoustic-to-articulatory inversion approach.…”

Section: Introductionmentioning

confidence: 77%

“…Their difficulty in using keyboard, mouse and touch screen based user interfaces makes speech controlled assistive technologies more natural alternatives [7], even though speech quality is degraded. Despite the rapid progress of automatic speech recognition (ASR) technologies in the past few decades, recognition of disordered speech is still a very challenging task to date due to severe mismatch against normal speech, difficulty in large scale data collection for system development and high level of variability among speakers [8][9][10][11][12][13].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition

Hu¹,

Liu²,

Xie³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Articulatory features are inherently invariant to acoustic signal distortion and have been successfully incorporated into automatic speech recognition (ASR) systems for normal speech. Their practical application to disordered speech recognition is often limited by the difficulty in collecting such specialist data from impaired speakers. This paper presents a cross-domain acousticto-articulatory (A2A) inversion approach that utilizes the parallel acoustic-articulatory data of the 15-hour TORGO corpus in model training before being cross-domain adapted to the 102.7-hour UASpeech corpus and to produce articulatory features. Mixture density networks based neural A2A inversion models were used. A cross-domain feature adaptation network was also used to reduce the acoustic mismatch between the TORGO and UASpeech data. On both tasks, incorporating the A2A generated articulatory features consistently outperformed the baseline hybrid DNN/TDNN, CTC and Conformer based end-to-end systems constructed using acoustic features only. The best multi-modal system incorporating video modality and the cross-domain articulatory features as well as data augmentation and learning hidden unit contributions (LHUC) speaker adaptation produced the lowest published word error rate (WER) of 24.82% on the 16 dysarthric speakers of the benchmark UASpeech task.

show abstract

Section: Introductionmentioning

confidence: 77%

Section: Introductionmentioning

confidence: 99%

Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition

Hu¹,

Liu²,

Xie³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Kim et al, 2016;Seong et al, 2016) and a deep learning model (Xiong et al, 2018). Overcoming the limitations of dysarthric speech as training data, researchers (1) used models that require less training data (Gemmeke et al, 2014), (2) augment data by artificially generating dysarthric speech (Green et al, 2021;Jin et al, 2021;Ko et al, 2017;Liu et al, 2021;Mariya Celin et al, 2020;Vachhani et al, 2018;Xiong et al, 2019), and (3) adapt data to a given speaker (Geng et al, 2021;Takashima et al, 2020). Further, Sriranjani et al (2015) used "data pooling," in which normal speech recordings were pooled from databases and combined with dysarthric speech data to train systems.…”

Section: Rq1: How the Characteristics Of Users' Speech Affect Their I...mentioning

confidence: 99%

Interaction between people with dysarthria and speech recognition systems: A review

Jaddoh

Loizides

Rana

2022

Assistive Technology

View full text Add to dashboard Cite

In recent years, rapid advancements have taken place for automatic speech recognition (ASR) systems and devices. Though ASR technologies have increased, the accessibility of these novel interaction systems is underreported and may present difficulties for people with speech impediments. In this article, we attempt to identify gaps in current research on the interaction between people with dysarthria and ASR systems and devices. We cover the period from 2011, when Siri (the first and the leading commercial voice assistant) was launched, to 2020. The review employs an interaction framework in which each element (user, input, system, and output) contributes to the interaction process. To select the articles for review, we conducted a search of scientific databases and academic journals. A total of 36 studies met the inclusion criteria, which included use of the word error rate (WER) as a measurement for evaluating ASR systems. This review determines that challenges in interacting with ASR systems persist even in light of the most recent commercial technologies. Further, understanding of the entire interaction process remains limited; thus, to improve this interaction, the recent progress of ASR systems must be elucidated.

show abstract

“…To this end, this paper investigates a novel set of techniques to incorporate speech impairment severity into state-ofthe-art hybrid DNN [13], end-to-end Conformer [37] and selfsupervised learning (SSL) based pre-trained Wav2vec 2.0 [38] ASR systems. These include the use of: a) multi-task [39] training cost interpolation between the ASR loss and speech impairment severity prediction error; b) spectral basis embedding (SBE) [11,25] based speaker-severity aware adaptation features that are trained to simultaneously predict both speaker-identity and impairment severity; and c) structured learning hidden units contribution (LHUC) [40] transforms that are separately conditioned on speaker-identity and impairment severity. These are used to facilitate both speaker-severity adaptive training of ASR systems and their test-time unsupervised adaptation to both factors of variability.…”

Section: Introductionmentioning

confidence: 99%

Confidence Score Based Conformer Speaker Adaptation for Speech Recognition

Xie¹,

Wang²,

Cui³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

A key challenge in dysarthric speech recognition is the speaker-level diversity attributed to both speaker-identity associated factors such as gender, and speech impairment severity. Most prior researches on addressing this issue focused on using speaker-identity only. To this end, this paper proposes a novel set of techniques to use both severity and speaker-identity in dysarthric speech recognition: a) multitask training incorporating severity prediction error; b) speaker-severity aware auxiliary feature adaptation; and c) structured LHUC transforms separately conditioned on speaker-identity and severity. Experiments conducted on UASpeech suggest incorporating additional speech impairment severity into state-of-the-art hybrid DNN, E2E Conformer and pre-trained Wav2vec 2.0 ASR systems produced statistically significant WER reductions up to 4.78% (14.03% relative). Using the best system the lowest published WER of 17.82% (51.25% on very low intelligibility) was obtained on UASpeech.

show abstract

Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition

Cited by 11 publications

References 38 publications

Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition

Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition

Interaction between people with dysarthria and speech recognition systems: A review

Confidence Score Based Conformer Speaker Adaptation for Speech Recognition

Contact Info

Product

Resources

About