Visyllable Based Speech Animation

Kshirsagar, Sumedha; Magnenat‐Thalmann, Nadia

doi:10.1111/1467-8659.t01-2-00711

Cited by 52 publications

(34 citation statements)

References 11 publications

(14 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The temporal aspects of facial performances are very important for synthesis of new facial animations from speech [Bregler et al 1997;Brand 1999;Ezzat et al 2002;Kshirsagar and Thalmann 2003;Cao et al 2004;Ma et al 2004;Deng et al 2005]. Most of these approaches record facial motion of speaking subjects and then recombine the recorded facial motion from learned parametric models to synthesize new facial motion.…”

Section: Related Workmentioning

confidence: 99%

Facial performance enhancement using dynamic shape space analysis

et al. 2014

View full text Add to dashboard Cite

The facial performance of an individual is inherently rich in subtle deformation and timing details. Although these subtleties make the performance realistic and compelling, they often elude both motion capture and hand animation. We present a technique for adding fine-scale details and expressiveness to low-resolution art-directed facial performances, such as those created manually using a rig, via marker-based capture, by fitting a morphable model to a video, or through Kinect reconstruction using recent faceshift technology. We employ a high-resolution facial performance capture system to acquire a representative performance of an individual in which he or she explores the full range of facial expressiveness. From the captured data, our system extracts an expressiveness model that encodes subtle spatial and temporal deformation details specific to that particular individual. Once this model has been built, these details can be transferred to low-resolution artdirected performances. We demonstrate results on various forms of input; after our enhancement, the resulting animations exhibit the same nuances and fine spatial details as the captured performance, with optional temporal enhancement to match the dynamics of the actor. Finally, we show that our technique outperforms the current state-of-the-art in example-based facial animation.

show abstract

Section: Related Workmentioning

confidence: 99%

Facial performance enhancement using dynamic shape space analysis

et al. 2014

View full text Add to dashboard Cite

show abstract

“…The motion synthesis is mainly on the concatenation of visual units. [2][3][4]6,7 Ezzat et al 2 employ a variant of Multidimensional Morphable Model (MMM) to embed mouth configurations. HMM is employed to model the probabilistic state machine for speech animation.…”

Section: Related Workmentioning

confidence: 99%

“…Currently, the 3D facial motions can be captured real-time with opticalbased devices. Given the large volume of dataset, most 3D speech motion synthesis algorithms are based on the concatenation of captured motion segments and the variable transition models [1][2][3][4][5] to handle the coarticulation. Hitherto, little efforts addressed the inside dynamic nature of the pronunciation unit, e.g., what dynamics inside the visyllable is and how the facial shape varies throughout one visyllable.…”

Section: Introductionmentioning

confidence: 99%

Stylized synthesis of facial speech motions

Pei

Zha²

2007

Computer Animation & Virtual

View full text Add to dashboard Cite

Stylized synthesis of facial speech motions is central to facial animation. Most synthesis algorithms put emphasis on the reasonable concatenation of captured motion segments. The dynamic modeling of speech units, e.g. visemes and visyllables (the visual appearance of a syllable), has not drawn much attention. In this paper, we address the fundamental issues regarding the stylized dynamic modeling of visyllables. The decomposable generalized model is learnt for the stylized motion synthesis. The visyllable modeling includes two parts: (1) A dynamic model for each kind of visyllable that is learnt based on a Gaussian Process Dynamical Model; (2) A multilinear model based unified mapping between the high dimensional observation space and low dimensional latent space. The dynamic visyllable model embeds the high dimensional motion data, and constructs the dynamic mapping in the latent space simultaneously. To generalize the visyllable model from several instances, the mapping coefficient matrices are assembled to a tensor, which is decomposed into independent modes, e.g. identity and uttering styles. Therefore, with the linear combination of components in each mode, the novel stylized motions can be synthesized.

show abstract

“…[Byun and Badler 2002] modified the MPEG-4 Facial Animation Parameters (FAPs) [Moving Picture Experts Group 1998] to add expressiveness, [Kshirsagar and Magnenat-Thalmann 2003] used PCA to deform the mouth during speech, and [Chai et al 2003] used facial tracking to drive animations from a motion capture database. [Wang et al 2004] used a multiresolution deformable mesh to track facial motion, and a low dimensional embedding technique to learn expression style.…”

Section: Related Workmentioning

confidence: 99%

Automatic determination of facial muscle activations from sparse motion capture marker data

2005

View full text Add to dashboard Cite

We built an anatomically accurate model of facial musculature, passive tissue and underlying skeletal structure using volumetric data acquired from a living male subject. The tissues are endowed with a highly nonlinear constitutive model including controllable anisotropic muscle activations based on fiber directions. Detailed models of this sort can be difficult to animate requiring complex coordinated stimulation of the underlying musculature. We propose a solution to this problem automatically determining muscle activations that track a sparse set of surface landmarks, e.g. acquired from motion capture marker data. Since the resulting animation is obtained via a three dimensional nonlinear finite element method, we obtain visually plausible and anatomically correct deformations with spatial and temporal coherence that provides robustness against outliers in the motion capture data. Moreover, the obtained muscle activations can be used in a robust simulation framework including contact and collision of the face with external objects.

show abstract

Visyllable Based Speech Animation

Cited by 52 publications

References 11 publications

Facial performance enhancement using dynamic shape space analysis

Facial performance enhancement using dynamic shape space analysis

Stylized synthesis of facial speech motions

Automatic determination of facial muscle activations from sparse motion capture marker data

Contact Info

Product

Resources

About