Tongue and Lip Motion Patterns in Alaryngeal Speech

Teplansky, Kristin; Wisler, Alan; Cao, Beiming; Liang, Wendy; Whited, Chad W.; Mau, Ted; Wang, Jun

doi:10.21437/interspeech.2020-2854

Cited by 7 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We observe that the syllable rate for the three speaking modes are significantly different at the level of p < .001. These results support those found in previous studies [20,21,22,23], indicating that silent speech is articulated slower than whispered speech, and that whispered speech is slower than modal speech.…”

Section: Discussionsupporting

confidence: 92%

“…for plosives or fricatives) causes speakers to produce incomplete or reduced movements, thus leading to hypo-articulation. Recent work has found evidence to support the latter hypothesis [20,21,22,23]. These studies have observed that silent speech exhibits longer duration and decreased articulatory activity when compared with modal speech.…”

Section: Introductionmentioning

confidence: 95%

“…Previous work has investigated and compared the articulatory properties of silent, whispered, and modal speaking modes. Studies have measured tongue and lip movements with EMA [19,20,21,22] or with ultrasound and video imaging [23]. Two conflicting hypotheses have arisen with regard to silent articulation [20].…”

Section: Introductionmentioning

confidence: 99%

“…However, evidence for hyper-articulation was also observed [19], which might occur only for specific speech sounds such as bilabial consonants. These studies relied on a relatively small number of participants, typically 4-12 speakers [7,21,22,19], with one study using 20 speakers [20].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Silent versus Modal Multi-Speaker Speech Recognition from Ultrasound and Video

et al. 2021

View full text Add to dashboard Cite

We investigate multi-speaker speech recognition from ultrasound images of the tongue and video images of the lips. We train our systems on imaging data from modal speech, and evaluate on matched test sets of two speaking modes: silent and modal speech. We observe that silent speech recognition from imaging data underperforms compared to modal speech recognition, likely due to a speaking-mode mismatch between training and testing. We improve silent speech recognition performance using techniques that address the domain mismatch, such as fMLLR and unsupervised model adaptation. We also analyse the properties of silent and modal speech in terms of utterance duration and the size of the articulatory space. To estimate the articulatory space, we compute the convex hull of tongue splines, extracted from ultrasound tongue images. Overall, we observe that the duration of silent speech is longer than that of modal speech, and that silent speech covers a smaller articulatory space than modal speech. Although these two properties are statistically significant across speaking modes, they do not directly correlate with word error rates from speech recognition.

show abstract

Section: Discussionsupporting

confidence: 92%

Section: Introductionmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Silent versus Modal Multi-Speaker Speech Recognition from Ultrasound and Video

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The spectral characteristics of the vowels speakers produce when they lack auditory feedback changes as well. It has also been shown that speakers take different strategies with respect to speaking rate and articulatory space when producing silent speech compared to voiced speech [16,17,18,19,20]. These observations all indicate the TaL speakers are likely to have articulated differently when speaking without auditory feedback.…”

Section: Silent Speech Versus Normally-voiced Speechmentioning

confidence: 99%

Silent Speech Recognition with Articulator Positions Estimated from Tongue Ultrasound and Lip Video

Beeson,

Richmond

2023

Interspeech 2023

View full text Add to dashboard Cite

We present a multi-speaker silent speech recognition system trained on articulator features derived from the Tongue and Lips corpus, a multi-speaker corpus of ultrasound tongue imaging and lip video data. We extracted articulator features using the pose estimation software DeepLabCut, then trained recognition models with these point-tracking features using Kaldi. We trained with voiced utterances, then tested performance on both voiced and silent utterances. Our multi-speaker SSR improved WER by 23.06% when compared to a previous similar multispeaker SSR system which used image-based instead of pointtracking features. We also found great improvements (up to 15.45% decrease in WER) in recognition of silent speech using fMLLR adaptation compared to raw features. Finally, we investigated differences in articulator trajectories between voiced and silent speech and found that speakers tend to miss articulatory targets that are present in voiced speech when speaking silently.

show abstract