Empirical modeling of human face kinematics during speech using motion clustering

Lucero, Jorge C.; Maciel, Susanne; Johns, Derek A.; Munhall, Kevin G.

doi:10.1121/1.1928807

Cited by 10 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Contrary to empirical models based on recorded data or medical images (for example Lucero et al (2005) or Badin et al (2008)), our approach in the last decade has been to develop biomechanical models of speech articulators, which are as close as possible to the human anatomy and functional morphology. Special emphasis has been given to the representation of the muscular structures and the rheological properties of soft tissues.…”

Section: Introductionmentioning

confidence: 99%

Simulation of dynamic orofacial movements using a constitutive law varying with muscle activation

Nazari

Perrier

Chabanas

et al. 2010

Computer Methods in Biomechanics and Biomedical Engineering

View full text Add to dashboard Cite

This paper presents a biomechanical model of the face to simulate orofacial movements in speech and non-verbal communication. A 3D finite element model, based on medical images of a subject, is presented. A hyperelastic Mooney-Rivlin constitutive law accounts for the non-linear behaviour of facial tissue. Muscle fibres are represented by piece-wise uniaxial tensile element that generate force. The stress stiffening effect, an increase in the stiffness of the muscles when activated, is modelled by varying the constitutive law of the tissue with the level of activation of the muscle. A large number of facial movements occurring during speech and facial mimics are simulated. Results show that our modelling approach provides a realistic account of facial mimics. The differences between dynamic vs. quasi-static simulations are also discussed, proving that dynamic trajectories better fit experimental data.

show abstract

Section: Introductionmentioning

confidence: 99%

Simulation of dynamic orofacial movements using a constitutive law varying with muscle activation

Nazari

Perrier

Chabanas

et al. 2010

Computer Methods in Biomechanics and Biomedical Engineering

View full text Add to dashboard Cite

show abstract

“…Experiment 1 was designed to establish that perceivers in fact selectively attend to the lip movements of the model speaker during visual vowel discrimination tasks that elicit perceptual biases. Detailed analyses of intramuscular recordings and kinematic data indicate that visible speech movements are spatially distributed across the whole face (e.g., Vatikiotis-Bateson, Munhall, Kasahara, et al, 1996; Vatikiotis-Bateson, Munhall, Hirayama, Lee, & Terzopolous, 1996; Munhall & Vatikiotis-Bateson, 2004; Lucero & Munhall, 1999; Lucero, Maciel, Johns, & Munhall, 2005). Furthermore, perception studies using eye-tracking methodology indicate that perceivers frequently gaze at facial regions that extend beyond the lips while they watch and listen to a speaker talk (e.g., Vatikiotis-Bateson, Eigsti, Yano, & Munhall, 1998; Paré, Richler, ten Hove, & Munhall, 2003; Everdell, Marsh, Yurick, Munhall, & Paré, 2007; Irwin & Brancazio, 2014).…”

mentioning

confidence: 99%

Asymmetries in unimodal visual vowel perception: The roles of oral-facial kinematics, orientation, and configuration.

Masapollo¹,

Polka²,

Ménard³

et al. 2018

Journal of Experimental Psychology: Human Perception and Perfor

View full text Add to dashboard Cite

Masapollo, Polka, and Ménard (2017) recently reported a robust directional asymmetry in unimodal visual vowel perception: Adult perceivers discriminate a change from an English /u/ viseme to a French /u/ viseme significantly better than a change in the reverse direction. This asymmetry replicates a frequent pattern found in unimodal auditory vowel perception that points to a universal bias favoring more extreme vocalic articulations, which lead to acoustic signals with increased formant convergence. In the present article, the authors report 5 experiments designed to investigate whether this asymmetry in the visual realm reflects a speech-specific or general processing bias. They successfully replicated the directional effect using Masapollo et al.'s dynamically articulating faces but failed to replicate the effect when the faces were shown under static conditions. Asymmetries also emerged during discrimination of canonically oriented point-light stimuli that retained the kinematics and configuration of the articulating mouth. In contrast, no asymmetries emerged during discrimination of rotated point-light stimuli or Lissajou patterns that retained the kinematics, but not the canonical orientation or spatial configuration, of the labial gestures. These findings suggest that the perceptual processes underlying asymmetries in unimodal visual vowel discrimination are sensitive to speech-specific motion and configural properties and raise foundational questions concerning the role of specialized and general processes in vowel perception. (PsycINFO Database Record

show abstract

“…While slower modulations were not found in the analysis of the GRID data, the GRID data revealed a number of more detailed orofacial components. Decomposition of the face during speech has been pursued in previous work using PCA [ 18 , 19 , 37 ], ICA [ 79 ] or other matrix factorization algorithms [ 38 , 40 ]. Lucero et al (2008) identified independent kinematic components for the upper and lower parts of the mouth and the two mouth corners, that were also identified in our AV analysis of the GRID data [ 40 ].…”

Section: Discussionmentioning

confidence: 99%

Modulation transfer functions for audiovisual speech

et al. 2022

View full text Add to dashboard Cite

Temporal synchrony between facial motion and acoustic modulations is a hallmark feature of audiovisual speech. The moving face and mouth during natural speech is known to be correlated with low-frequency acoustic envelope fluctuations (below 10 Hz), but the precise rates at which envelope information is synchronized with motion in different parts of the face are less clear. Here, we used regularized canonical correlation analysis (rCCA) to learn speech envelope filters whose outputs correlate with motion in different parts of the speakers face. We leveraged recent advances in video-based 3D facial landmark estimation allowing us to examine statistical envelope-face correlations across a large number of speakers (∼4000). Specifically, rCCA was used to learn modulation transfer functions (MTFs) for the speech envelope that significantly predict correlation with facial motion across different speakers. The AV analysis revealed bandpass speech envelope filters at distinct temporal scales. A first set of MTFs showed peaks around 3-4 Hz and were correlated with mouth movements. A second set of MTFs captured envelope fluctuations in the 1-2 Hz range correlated with more global face and head motion. These two distinctive timescales emerged only as a property of natural AV speech statistics across many speakers. A similar analysis of fewer speakers performing a controlled speech task highlighted only the well-known temporal modulations around 4 Hz correlated with orofacial motion. The different bandpass ranges of AV correlation align notably with the average rates at which syllables (3-4 Hz) and phrases (1-2 Hz) are produced in natural speech. Whereas periodicities at the syllable rate are evident in the envelope spectrum of the speech signal itself, slower 1-2 Hz regularities thus only become prominent when considering crossmodal signal statistics. This may indicate a motor origin of temporal regularities at the timescales of syllables and phrases in natural speech.

show abstract

Empirical modeling of human face kinematics during speech using motion clustering

Cited by 10 publications

References 11 publications

Simulation of dynamic orofacial movements using a constitutive law varying with muscle activation

Simulation of dynamic orofacial movements using a constitutive law varying with muscle activation

Asymmetries in unimodal visual vowel perception: The roles of oral-facial kinematics, orientation, and configuration.

Modulation transfer functions for audiovisual speech

Contact Info

Product

Resources

About