A Machine Learning Approach to Tongue Motion Analysis in 2D Ultrasound Image Sequences

Abstract

Abstract. Analysis of tongue motions as captured in dynamic ultrasound (US) images has been an important tool in speech research. Previous studies generally required semi-automatic tongue segmentations to perform data analysis. In this paper, we adopt a machine learning approach that does not require tongue segmentation. Specifically, we employ advanced normalization procedures to temporally register the US sequences using their corresponding audio files. To explicitly encode motion, we then register the image frames spatio-temporally to compute a set of deformation fields from which we construct the velocity-based and spatiotemporal gestural descriptors, where the latter explicitly encode tongue dynamics during speech. Next, making use of the recently proposed Histogram Intersection Kernel, we perform support vector machine classification to evaluate the extracted descriptors with a set of clinical measures. We applied our method to speech abnormality and tongue gestures prediction. Overall, differentiating tongue motion, as produced by patients with or without speech impediments on a dataset of 24 US sequences, was achieved with classification accuracy of 94%. When applied to another dataset of 90 US sequences for two other classification tasks, accuracies were 86% and 84%.

Fully-automated tongue detection in ultrasound images

“…These advantages of US facilitate articulatory gains in speech therapy for many patients (Bernhardt et al (2005), Shawker & Sonies (1985), Bacsfalvi & Bernhardt (2011)). US imaging is one of the standard routines in the phonetics literature (Chi-Fishman (2005), Li et al (2005a), Tang et al (2011)), this thesis also focuses on detection of tongue contours from US images.…”

Differentiating post-cancer from healthy tongue muscle coordination patterns during speech using deep learning

“…In the context of tongue motion analysis for speech using various imaging and motion capture techniques, several researchers have studied the classification of speech movements using machine learning. For example, Tang et al (2011) investigated spatiotemporal gestural descriptors from ultrasound for classifying speech movements using a support vector machine (SVM). Wang et al (2016) examined a set of flesh points on the tongue and lips that are optimal to classify speech movements via a SVM from electromagnetic articulographs.…”