Labial coarticulation modeling for realistic facial animation

Cosi, Piero; Caldognetto, Emanuela Magno; Perin, Giorgio; Zmarich, Claudio

doi:10.1109/icmi.2002.1167047

Cited by 29 publications

(15 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As explained in [6][7][8], several Audio/Visual corpora, were used to train our MPEG-4 [9] standard talking head called LUCIA [10] speaking with an Italian version of FESTIVAL TTS [11].…”

Section: Data-driven Methodology and Toolsmentioning

confidence: 99%

“…where F(n) is generated by a modified version of the Cohen-Massaro co-articulation model [13] as introduced in [6][7]. Even if the number of parameters to be optimized is rather high, the size of the data corpus is large enough to allow a meaningful estimation, but, due to the presence of several local minima, the optimization process has to be manually controlled in order to assist the algorithm convergence.…”

Section: Model Estimationmentioning

confidence: 99%

See 1 more Smart Citation

Data-Driven Tools for Designing Talking Heads Exploiting Emotional Attitudes

Cosi

Fusaro

Grigoletto

et al. 2004

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…As explained in [6][7][8], several Audio/Visual corpora, were used to train our MPEG-4 [9] standard talking head called LUCIA [10] speaking with an Italian version of FESTIVAL TTS [11].…”

Section: Data-driven Methodology and Toolsmentioning

confidence: 99%

Section: Model Estimationmentioning

confidence: 99%

Data-Driven Tools for Designing Talking Heads Exploiting Emotional Attitudes

Cosi

Fusaro

Grigoletto

et al. 2004

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…the time-locked or look-ahead model) that are difficult to explain by the simpler diphone or triphone model. Their methods are later extended by Cosi et al [38] with the resistance functions and shape functions, which is the basic concept of the animeme.…”

Section: B Lip-sync Speech Animationmentioning

confidence: 99%

Animating Lip-Sync Characters With Dominated Animeme Models

Chen

Huang

Guan

et al. 2012

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Abstract-Character speech animation is traditionally considered as important but tedious work, especially when taking lip synchronization (lip-sync) into consideration. Although there are some methods proposed to ease the burden on artists to create facial and speech animation, almost none are fast and efficient. In this paper, we introduce a framework for synthesizing lip-sync character speech animation in real time from a given speech sequence and its corresponding texts. Starting from training dominated animeme models for each kind of phoneme by learning the character's animation control signal through an EM-style optimization approach. The dominated animeme models are further decomposed to polynomial-fitted animeme models and corresponding dominance functions while taking coarticulation into account. Finally, given a novel speech sequence and its corresponding texts, the animation control signal of the character can be synthesized in real time with the trained dominated animeme models. The synthesized lip-sync animation can even preserve exaggerated characteristics of the character's facial geometry. Moreover, since our method can perform in real time, it can be used for many applications, such as lip-sync animation prototyping, multilingual animation reproduction, avatar speech, mass animation production, etc. Furthermore, the synthesized animation control signal can still be imported into 3D packages for further adjustment, so our method can be easily integrated into existing production pipeline.

show abstract

“…The early work by Cohen et al introduced dominance functions [Cohen and Massaro 1993;Massaro et al 2012] as a parameterization method to deal with co-articulation. Their pioneering work is followed by more research works aimed to improve the dominance function model [Cosi et al 2002;King and Parent 2005;Cohen et al 2002]. In their work each parameter curve controls a time-varying deformation over a small region on the face model.…”

Section: Visual Speech Animationmentioning

confidence: 99%

A Practical and Configurable Lip Sync Method for Games

Feng

Marsella

et al. 2013

Proceedings of Motion on Games

View full text Add to dashboard Cite

We demonstrate a lip animation (lip sync) algorithm for real-time applications that can be used to generate synchronized facial movements with audio generated from natural speech or a text-to-speech engine. Our method requires an animator to construct animations using a canonical set of visemes for all pairwise combinations of a reduced phoneme set (phone bigrams). These animations are then stitched together to construct the final animation, adding velocity and lip-pose constraints. This method can be applied to any character that uses the same, small set of visemes. Our method can operate efficiently in multiple languages by reusing phone bigram animations that are shared among languages, and specific word sounds can be identified and changed on a per-character basis. Our method uses no machine learning, which offers two advantages over techniques that do: 1) data can be generated for non-human characters whose faces can not be easily retargeted from a human speaker's face, and 2) the specific facial poses or shapes used for animation can be specified during the setup and rigging stage, and before the lip animation stage, thus making it suitable for game pipelines or circumstances where the speech targets poses are predetermined, such as after acquisition from an online 3D marketplace. Synchronizing the lip and mouth movements naturally along with animation is an important part of convincing 3D character performance. In this paper, we present a simple, portable and editable lip-synchronization method that works for multiple languages, requires no machine learning, can be constructed by a skilled animator, is effective for real-time simulations such as games, and can be personalized for each character. Our method associates animation curves designed by an animator on a fixed set of static facial poses, with sequential pairs of phonemes (phone bigrams), and then stitches these animations together to create a set of curves for the facial poses along with constraints that ensure that key poses are properly played. Diphone-and triphone-based methods have been explored in various previous works, often requiring machine learning. However, our experiments have shown that animating phoneme pairs (such as phone bigrams or diphones), as opposed to phoneme triples or longer sequences of phonemes, is sufficient for many types of animated characters. Also, our experiments have shown that skilled animators can sufficiently generate the data needed for good quality results. Thus our algorithm does not need any specific rules about coarticulation, such as dominance functions or language rules. Such rules are implicit within the artist-produced data. In order to produce a tractible set of data, our method reduces

show abstract

Labial coarticulation modeling for realistic facial animation

Cited by 29 publications

References 8 publications

Data-Driven Tools for Designing Talking Heads Exploiting Emotional Attitudes

Data-Driven Tools for Designing Talking Heads Exploiting Emotional Attitudes

Animating Lip-Sync Characters With Dominated Animeme Models

A Practical and Configurable Lip Sync Method for Games

Contact Info

Product

Resources

About