A Practical and Configurable Lip Sync Method for Games

Abstract

We demonstrate a lip animation (lip sync) algorithm for real-time applications that can be used to generate synchronized facial movements with audio generated from natural speech or a text-to-speech engine. Our method requires an animator to construct animations using a canonical set of visemes for all pairwise combinations of a reduced phoneme set (phone bigrams). These animations are then stitched together to construct the final animation, adding velocity and lip-pose constraints. This method can be applied to any character that uses the same, small set of visemes. Our method can operate efficiently in multiple languages by reusing phone bigram animations that are shared among languages, and specific word sounds can be identified and changed on a per-character basis. Our method uses no machine learning, which offers two advantages over techniques that do: 1) data can be generated for non-human characters whose faces can not be easily retargeted from a human speaker's face, and 2) the specific facial poses or shapes used for animation can be specified during the setup and rigging stage, and before the lip animation stage, thus making it suitable for game pipelines or circumstances where the speech targets poses are predetermined, such as after acquisition from an online 3D marketplace. Synchronizing the lip and mouth movements naturally along with animation is an important part of convincing 3D character performance. In this paper, we present a simple, portable and editable lip-synchronization method that works for multiple languages, requires no machine learning, can be constructed by a skilled animator, is effective for real-time simulations such as games, and can be personalized for each character. Our method associates animation curves designed by an animator on a fixed set of static facial poses, with sequential pairs of phonemes (phone bigrams), and then stitches these animations together to create a set of curves for the facial poses along with constraints that ensure that key poses are properly played. Diphone-and triphone-based methods have been explored in various previous works, often requiring machine learning. However, our experiments have shown that animating phoneme pairs (such as phone bigrams or diphones), as opposed to phoneme triples or longer sequences of phonemes, is sufficient for many types of animated characters. Also, our experiments have shown that skilled animators can sufficiently generate the data needed for good quality results. Thus our algorithm does not need any specific rules about coarticulation, such as dominance functions or language rules. Such rules are implicit within the artist-produced data. In order to produce a tractible set of data, our method reduces

A deep learning approach for generalized speech animation

“…words or phonemes) [Bregler et al 1997;Cao et al 2005;Liu and Ostermann 2012;Mattheyses et al 2013;Xu et al 2013] or variable length [Cosatto and Graf 2000;Edwards et al 2016;Ma et al 2006;Taylor et al 2012] units. Unit selection typically involves minimizing a cost function based on the phonetic context and the smoothness.…”

Lip syncing method for realistic expressive 3D face model

“…Better results can be obtained by animating or learning combinations of three phonemes or even longer sequences. Figure 19 illustrates the architecture of lip syncing method following Quieruze [31] and Ari Shariro [44]. Basically, this method receives a TTS file or Audio file as input containing the character's speech and an auxiliary file containing its textual description.…”

Virtual humans for training and assessment of self-medication consultation skills in pharmacy students

“…Resorting to a Microsoft speech library [13], the Speech Generator module creates two files: the audio data file and the animation data file which are used by the Speech Animation Controller to synchronize the audio with the blend shapes animation.…”