Spatio-temporal articulatory movement primitives during speech production: Extraction, interpretation, and validation

Ramanarayanan, Vikram; Goldstein, Louis; Narayanan, Shrikanth

doi:10.1121/1.4812765

Cited by 32 publications

(43 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They derive these multiplicative updates by choosing an adaptive learning rate that makes additive terms cancel from standard gradient descent on the cost function. We will reproduce their derivation here, and detail how to extend it to the convolutional case [41] apply several forms of regularization [30,36,7]. See Table 2 for a compilation of cost functions, derivatives and multiplicative updates for NMF and CNMF under several different regularization conditions.…”

Section: Deriving Multiplicative Update Rulesmentioning

confidence: 99%

Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience

Mackevicius

Bahle

Williams

et al. 2018

Preprint

View full text Add to dashboard Cite

2The ability to identify interpretable, low-dimensional features that capture the dynamics 3 of large-scale neural recordings is a major challenge in neuroscience. Dynamics that 4 include repeated temporal patterns (which we call sequences), are not succinctly 5 captured by traditional dimensionality reduction techniques such as principal 6 components analysis (PCA) and non-negative matrix factorization (NMF). The presence of 7 neural sequences is commonly demonstrated using visual display of trial-averaged firing 8 rates [15, 32,19]. However, the field suffers from a lack of task-independent, 9 unsupervised tools for consistently identifying sequences directly from neural data, and 10 cross-validating these sequences on held-out data. We propose a tool that extends a 11 convolutional NMF technique to prevent its common failure modes. Our method, which 12 we call seqNMF, provides a framework for extracting sequences from a dataset, and is 13 easily cross-validated to assess the significance of each extracted factor. We apply 14 seqNMF to recover sequences in both a previously published dataset from rat 15 hippocampus, as well as a new dataset from the songbird pre-motor area, HVC. In the 16 hippocampal data, our algorithm automatically identifies neural sequences that match was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.

show abstract

Section: Deriving Multiplicative Update Rulesmentioning

confidence: 99%

Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience

Mackevicius

Bahle

Williams

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Techniques such as Electromagnetic Articulography, offering excellent temporal resolution, and Magnetic Resonance Imaging, offering superior spatial resolution, have enabled a much better analysis of the speech production process (Ramanarayanan et al, 2013). Many insights have thus been gained regarding the spatio-temporal details of speech generation.…”

Section: Robust Feature Extractionmentioning

confidence: 99%

State of the art in statistical methods for language and speech processing

Bellegarda

Monz

2016

Computer Speech & Language

View full text Add to dashboard Cite

Recent years have seen rapid growth in the deployment of statistical methods for computational language and speech processing. The current popularity of such methods can be traced to the convergence of several factors, including the increasing amount of data now accessible, sustained advances in computing power and storage capabilities, and ongoing improvements in machine learning algorithms. The purpose of this contribution is to review the state of the art in both areas, point out the top trends in statistical modelling across a wide range of problems, and identify their most salient characteristics. The paper concludes with some prognostications regarding the likely impact on the field going forward.

show abstract

“…These models are usually obtained by Magnetic Resonance Imaging (Badin et al 2002; Engwall 2000; Badin et al 2008) and animated by electromagnetic articulography data (Engwall 2003; Gibert et al 2012; Steiner et al 2013) or ultrasound images (Fabre et al 2014). Computational approaches such as convolutive Nonnegative Matrix Factorization could be used to derive interpretable movement primitives from speech production data (Ramanarayanan et al 2013). These speech movement primitives can be used to animate virtual agents’ speech articulators for a given set of activation data.…”

Section: Introductionmentioning

confidence: 99%

Transforming an embodied conversational agent into an efficient talking head: from keyframe-based animation to multimodal concatenation synthesis

et al. 2015

View full text Add to dashboard Cite

BackgroundVirtual humans have become part of our everyday life (movies, internet, and computer games). Even though they are becoming more and more realistic, their speech capabilities are, most of the time, limited and not coherent and/or not synchronous with the corresponding acoustic signal.MethodsWe describe a method to convert a virtual human avatar (animated through key frames and interpolation) into a more naturalistic talking head. In fact, speech articulation cannot be accurately replicated using interpolation between key frames and talking heads with good speech capabilities are derived from real speech production data. Motion capture data are commonly used to provide accurate facial motion for visible speech articulators (jaw and lips) synchronous with acoustics. To access tongue trajectories (partially occluded speech articulator), electromagnetic articulography (EMA) is often used. We recorded a large database of phonetically-balanced English sentences with synchronous EMA, motion capture data, and acoustics. An articulatory model was computed on this database to recover missing data and to provide ‘normalized’ animation (i.e., articulatory) parameters. In addition, semi-automatic segmentation was performed on the acoustic stream. A dictionary of multimodal Australian English diphones was created. It is composed of the variation of the articulatory parameters between all the successive stable allophones.ResultsThe avatar’s facial key frames were converted into articulatory parameters steering its speech articulators (jaw, lips and tongue). The speech production database was used to drive the Embodied Conversational Agent (ECA) and to enhance its speech capabilities. A Text-To-Auditory Visual Speech synthesizer was created based on the MaryTTS software and on the diphone dictionary derived from the speech production database.ConclusionsWe describe a method to transform an ECA with generic tongue model and animation by key frames into a talking head that displays naturalistic tongue, jaw and lip motions. Thanks to a multimodal speech production database, a Text-To-Auditory Visual Speech synthesizer drives the ECA’s facial movements enhancing its speech capabilities.Electronic supplementary materialThe online version of this article (doi:10.1186/s40469-015-0007-8) contains supplementary material, which is available to authorized users.

show abstract

Spatio-temporal articulatory movement primitives during speech production: Extraction, interpretation, and validation

Cited by 32 publications

References 39 publications

Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience

Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience

State of the art in statistical methods for language and speech processing

Transforming an embodied conversational agent into an efficient talking head: from keyframe-based animation to multimodal concatenation synthesis

Contact Info

Product

Resources

About