Co-registration of speech production datasets from electromagnetic articulography and real-time magnetic resonance imaging

Kim, Jangwon; Lammert, Adam C.; Ghosh, Prasanta Kumar; Narayanan, Shrikanth

doi:10.1121/1.4862880

Cited by 14 publications

(7 citation statements)

References 9 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Electromagnetic articulography provides only a spatially sparse representation of the tongue movements. Co-registration methods of EMA and real-time magnetic resonance imaging (rtMRI) data (Kim et al 2014) provides richer spatio-temporal data to animate the tongue movements and create a better 3D tongue model. The proposed method could be applied on the USC-TIMIT (Narayanan et al 2014) which is an extensive database of multimodal (EMA, rtMRI, acoustics) speech production.…”

Section: Discussionmentioning

confidence: 99%

Transforming an embodied conversational agent into an efficient talking head: from keyframe-based animation to multimodal concatenation synthesis

et al. 2015

View full text Add to dashboard Cite

BackgroundVirtual humans have become part of our everyday life (movies, internet, and computer games). Even though they are becoming more and more realistic, their speech capabilities are, most of the time, limited and not coherent and/or not synchronous with the corresponding acoustic signal.MethodsWe describe a method to convert a virtual human avatar (animated through key frames and interpolation) into a more naturalistic talking head. In fact, speech articulation cannot be accurately replicated using interpolation between key frames and talking heads with good speech capabilities are derived from real speech production data. Motion capture data are commonly used to provide accurate facial motion for visible speech articulators (jaw and lips) synchronous with acoustics. To access tongue trajectories (partially occluded speech articulator), electromagnetic articulography (EMA) is often used. We recorded a large database of phonetically-balanced English sentences with synchronous EMA, motion capture data, and acoustics. An articulatory model was computed on this database to recover missing data and to provide ‘normalized’ animation (i.e., articulatory) parameters. In addition, semi-automatic segmentation was performed on the acoustic stream. A dictionary of multimodal Australian English diphones was created. It is composed of the variation of the articulatory parameters between all the successive stable allophones.ResultsThe avatar’s facial key frames were converted into articulatory parameters steering its speech articulators (jaw, lips and tongue). The speech production database was used to drive the Embodied Conversational Agent (ECA) and to enhance its speech capabilities. A Text-To-Auditory Visual Speech synthesizer was created based on the MaryTTS software and on the diphone dictionary derived from the speech production database.ConclusionsWe describe a method to transform an ECA with generic tongue model and animation by key frames into a talking head that displays naturalistic tongue, jaw and lip motions. Thanks to a multimodal speech production database, a Text-To-Auditory Visual Speech synthesizer drives the ECA’s facial movements enhancing its speech capabilities.Electronic supplementary materialThe online version of this article (doi:10.1186/s40469-015-0007-8) contains supplementary material, which is available to authorized users.

show abstract

Section: Discussionmentioning

confidence: 99%

Transforming an embodied conversational agent into an efficient talking head: from keyframe-based animation to multimodal concatenation synthesis

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Multimodality -data provided by different modalities and concerning similar phenomena or providing complementary data might benefit from joint analysis. For example, several technologies that support speech production studies (e.g., EMA, Kim et al 2014 andultrasound, Laprie et al 2014) are used in combination with MRI (Scott et al, 2014). Regardless of how the different data is analysed, if their individual contributions to the understanding of specific phenomena could be gathered in joint representations it might motivate a generalization of multimodal studies and an easier interpretation of the data.…”

Section: Challengesmentioning

confidence: 99%

Quantitative systematic analysis of vocal tract data

Silva

Teixeira

2016

Computer Speech & Language

View full text Add to dashboard Cite

show abstract

“…In addition, tagged-MRI (Parthasarathy et al 2007) allows us to observe internal tissue point motion, thereby detailing our understanding of the role of internal muscles during speech. Further, recent advances in various MRI methods have accelerated new advances in image and motion analyses such as segmentation of the tongue (Harandi et al 2014; Lee et al 2014) and internal muscles (Ibragimov et al 2015), internal motion tracking (Parthasarathy et al 2007), motion clustering (Woo et al 2014), and registration (Woo et al 2015c; Kim et al 2014) for various applications.…”

Section: Introductionmentioning

confidence: 99%

A spatio-temporal atlas and statistical model of the tongue during speech from cine-MRI

Woo

Xing

Lee

et al. 2016

Computer Methods in Biomechanics and Biomedical Engineering: Im

View full text Add to dashboard Cite

Statistical modeling of tongue motion during speech using cine magnetic resonance imaging (MRI) provides key information about the relationship between structure and motion of the tongue. In order to study the variability of tongue shape and motion in populations, a consistent integration and characterization of inter-subject variability is needed. In this paper, a method to construct a spatio-temporal atlas comprising a mean motion model and statistical modes of variation during speech is presented. The model is based on the cine-MRI from twenty two normal speakers and consists of several steps involving both spatial and temporal alignment problems independently. First, all images are registered into a common reference space, which is taken to be a neutral resting position of the tongue. Second, the tongue shapes of each individual relative to this reference space are produced. Third, a time warping approach (several are evaluated) is used to align the time frames of each subject to a common time series of initial mean images. Finally, the spatio-temporal atlas is created by time-warping each subject, generating new mean images at each time, and producing shape statistics around these mean images using principal component analysis at each reference time frame. Experimental results consist of comparison of various parameters and methods in creation of the atlas and a demonstration of the final modes of variations at various key time frames in a sample phrase.

show abstract

Co-registration of speech production datasets from electromagnetic articulography and real-time magnetic resonance imaging

Cited by 14 publications

References 9 publications

Transforming an embodied conversational agent into an efficient talking head: from keyframe-based animation to multimodal concatenation synthesis

Transforming an embodied conversational agent into an efficient talking head: from keyframe-based animation to multimodal concatenation synthesis

Quantitative systematic analysis of vocal tract data

A spatio-temporal atlas and statistical model of the tongue during speech from cine-MRI

Contact Info

Product

Resources

About