Ultrasound Tongue Imaging for Diarization and Alignment of Child Speech Therapy Sessions

Abstract

We investigate the automatic processing of child speech therapy sessions using ultrasound visual biofeedback, with a specific focus on complementing acoustic features with ultrasound images of the tongue for the tasks of speaker diarization and time-alignment of target words. For speaker diarization, we propose an ultrasound-based time-domain signal which we call estimated tongue activity. For word-alignment, we augment an acoustic model with low-dimensional representations of ultrasound images of the tongue, learned by a convolutional neural network. We conduct our experiments using the Ultrasuite repository of ultrasound and speech recordings for child speech therapy sessions. For both tasks, we observe that systems augmented with ultrasound data outperform corresponding systems using only the audio signal.

Exploiting ultrasound tongue imaging for the automatic detection of speech articulation errors

“…These results meet our expectations that ultrasound and audio complement each other well and that additional out-of-domain training data is beneficial. Similar findings were reported on related tasks, such as speaker diarisation and word alignment of speech therapy sessions (Ribeiro et al, 2019b).…”

“…Additionally, U-VBF can contribute to the automatic processing of speech therapy recordings. Recent work used ultrasound data to develop tongue contour extractors (Fabre et al, 2015), animate a tongue model (Fabre et al, 2017), automatically synchronise therapy recordings (Eshky et al, 2019), and for speaker diarisation and alignment of therapy sessions (Ribeiro et al, 2019b). There are, however, several challenges associated with the automatic processing of ultrasound tongue images (Stone, 2005;Ribeiro et al, 2019a).…”

“…With the widespread use of technology, there is increas-ing interest in automatically processing speech therapy tasks. This type of automation can be helpful to teachers and parents, who may use screening tools to determine the presence or absence of SSDs (Sadeghian & Zahorian, 2015;Ward et al, 2016); and to clinicians, who can save time on these time-consuming tasks (Ribeiro et al, 2019b). Clinicians and researchers are trained to use instrumented methods such as spectrograms or ultrasound tongue imaging to assist in the assessment, diagnosis, or quantification of treatment efficacy.…”

Exploiting ultrasound tongue imaging for the automatic detection of speech articulation errors

“…Additionally, U-VBF can contribute to the automatic processing of speech therapy recordings. Recent work used ultrasound data to develop tongue contour extractors (Fabre et al, 2015), animate a tongue model (Fabre et al, 2017), automatically synchronise therapy recordings (Eshky et al, 2019), and for speaker diarisation and alignment of therapy sessions (Ribeiro et al, 2019b). There are, however, several challenges associated with the automatic processing of ultrasound tongue images (Stone, 2005;Ribeiro et al, 2019a).…”

TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos

“…Ultrasound tongue imaging has been used in various applications, including speech therapy [6,7,8], language learning [9,10], phonetics studies [4], and the development of silent speech interfaces [11]. Previous work in the context of speech therapy used ultrasound data to develop tongue contour extractors [12], animate a tongue model [13], and automatically synchronise and process speech therapy recordings [14,15]. Additionally, speech recognition [16,17] and speech synthesis [18,19] from ultrasound images have been used in silent speech interfaces to restore spoken communication for users with voice impairments or to allow silent communication in situations where audible speech is undesirable.…”