Engineering Innovation in Speech Science: Data and Technologies

Hagedorn, Christina; Sorensen, Tanner; Lammert, Adam C.; Toutios, Asterios; Goldstein, Louis; Byrd, Dani; Narayanan, Shrikanth

doi:10.1044/2018_pers-sig19-2018-0003

Cited by 9 publications

(3 citation statements)

References 57 publications

(69 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, real-time Magnetic Resonance Imaging (rtMRI) has been emerging as an important tool in understanding speech production [1]. The rtMRI of a speaker allows visualising the dynamics of various speech articulators, irrespective of language or health conditions.…”

Section: Introductionmentioning

confidence: 99%

Real-Time MRI Video Synthesis from Time Aligned Phonemes with Sequence-to-Sequence Networks

Udupa

Ghosh

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Real-Time Magnetic resonance imaging (rtMRI) of the midsagittal plane of the mouth is of interest for speech production research. In this work, we focus on estimating utterance level rtMRI video from the spoken phoneme sequence. We obtain time-aligned phonemes from forced alignment, to obtain frame-level phoneme sequences which are aligned with rtMRI frames. We propose a sequence-tosequence learning model with a transformer phoneme encoder and convolutional frame decoder. We then modify the learning by using intermediary features obtained from sampling from a pretrained phoneme-conditioned variational autoencoder (CVAE). We train on 8 subjects in a subject-specific manner and demonstrate the performance with a subjective test. We also use an auxiliary task of air tissue boundary (ATB) segmentation to obtain the objective scores on the proposed models. We show that the proposed method is able to generate realistic rtMRI video for unseen utterances, and adding CVAE is beneficial for learning the sequence-to-sequence mapping for subjects where the mapping is hard to learn.

show abstract

Section: Introductionmentioning

confidence: 99%

Real-Time MRI Video Synthesis from Time Aligned Phonemes with Sequence-to-Sequence Networks

Udupa

Ghosh

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Real-time Magnetic Resonance Imaging (rtMRI) is a tool used exhaustively in speech science and linguistics to understand the speech production process in depth across languages and health conditions [1]. rtMRI has two advantages over other methods like X-ray [2], Electromagnetic articulography [3] and Ultrasound [4] -it is safe and non-invasive, and it captures a complete picture of the vocal tract including pharyngeal structures [5].…”

Section: Introductionmentioning

confidence: 99%

An error correction scheme for improved air-tissue boundary in real-time MRI video for speech production

Roy¹,

Belagali²,

Ghosh³

2022

Preprint

View full text Add to dashboard Cite

The best performance in Air-tissue boundary (ATB) segmentation of real-time Magnetic Resonance Imaging (rtMRI) videos in speech production is known to be achieved by a 3-dimensional convolutional neural network (3D-CNN) model. However, the evaluation of this model, as well as other ATB segmentation techniques reported in the literature, is done using Dynamic Time Warping (DTW) distance between the entire original and predicted contours. Such an evaluation measure may not capture local errors in the predicted contour. Careful analysis of predicted contours reveals errors in regions like the velum part of contour1 (ATB comprising of upper lip, hard palate, and velum) and tongue base section of contour2 (ATB covering jawline, lower lip, tongue base, and epiglottis), which are not captured in a global evaluation metric like DTW distance. In this work, we automatically detect such errors and propose a correction scheme for the same. We also propose two new evaluation metrics for ATB segmentation separately in contour1 and contour2 to explicitly capture two types of errors in these contours. The proposed detection and correction strategies result in an improvement of these two evaluation metrics by 61.8% and 61.4% for contour1 and by 67.8% and 28.4% for contour2. Traditional DTW distance, on the other hand, improves by 44.6% for contour1 and 4.0% for contour2.

show abstract

“…speech production poses a multitude of open research questions with implications for linguistics and speech science, as well as clinical and technological applications, creating a demand for improved methods for observing and measuring the vocal instrument in action [3,4].…”

mentioning

confidence: 99%

A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images

Lim,

Toutios,

Bliesener

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Real-time magnetic resonance imaging (RT-MRI) of human speech production is enabling significant advances in speech science, linguistics, bioinspired speech technology development, and clinical applications. Easy access to RT-MRI is however limited, and comprehensive datasets with broad access are needed to catalyze research across numerous domains. The imaging of the rapidly moving articulators and dynamic airway shaping during speech demands high spatio-temporal resolution and robust reconstruction methods. Further, while reconstructed images have been published, to-date there is no open dataset providing raw multi-coil RT-MRI data from an optimized speech production experimental setup. Such datasets could enable new and improved methods for dynamic image reconstruction, artifact correction, feature extraction, and direct extraction of linguistically-relevant biomarkers. The present dataset offers a unique corpus of 2D sagittal-view RT-MRI videos along with synchronized audio for 75 subjects performing linguistically motivated speech tasks, alongside the corresponding first-ever public domain raw RT-MRI data. The dataset also includes 3D volumetric vocal tract MRI during sustained speech sounds and high-resolution static anatomical T2-weighted upper airway MRI for each subject.

show abstract

Engineering Innovation in Speech Science: Data and Technologies

Cited by 9 publications

References 57 publications

Real-Time MRI Video Synthesis from Time Aligned Phonemes with Sequence-to-Sequence Networks

Real-Time MRI Video Synthesis from Time Aligned Phonemes with Sequence-to-Sequence Networks

An error correction scheme for improved air-tissue boundary in real-time MRI video for speech production

A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images

Contact Info

Product

Resources

About