Speech driven realistic mouth animation based on multi-modal unit selection

Abstract

This paper presents a novel audio visual diviseme (viseme pair) instance selection and concatenation method for speech driven photo realistic mouth animation. Firstly, an audio visual diviseme database is built consisting of the audio feature sequences, intensity sequences and visual feature sequences of the instances. In the Viterbi based diviseme instance selection, we set the accumulative cost as the weighted sum of three items: 1) logarithm of concatenation smoothness of the synthesized mouth trajectory; 2) logarithm of the pronunciation distance; 3) logarithm of the audio intensity distance between the candidate diviseme instance and the target diviseme segment in the incoming speech. The selected diviseme instances are time warped and blended to construct the mouth animation. Objective and subjective evaluations on the synthesized mouth animations prove that the multimodal diviseme instance selection algorithm proposed in this paper outperforms the triphone unit selection algorithm in Video Rewrite. Clear, accurate, smooth mouth animations can be obtained matching well with the pronunciation and intensity changes in the incoming speech. Moreover, with the logarithm function in the accumulative cost, it is easy to set the weights to obtain optimal mouth animations.

Speech driven photo-realistic face animation with mouth and jaw dynamics

“…These mouth images are then used to construct an audio visual diviseme (viseme pair) unit database. For an input audio speech, the multi-modal diviseme unit selection method [8] is adopted by considering the smoothness of the synthesized mouth movements, as well as the similarity of intensity and pronunciation between the input speech and the diviseme unit. The mouth image sequences of the selected diviseme units are then time warped and concatenated.…”

“…According to the underlying face model, these systems can be categorized into 3D-model-based animation or image-based rendering systems. While 3D graphical models [1,2] provide parametric control but lack of video realism, image-based approaches [6][7][8] have the potential of achieving very high levels of video realism.…”

“…In [8], a speech driven mouth synthesis method based on multi-modal diviseme unit selection has been proposed. An audio visual diviseme database consisting of the acoustic feature and intensity sequences, as well the visual feature sequences of the diviseme instances, is first build.…”

“…The image sequences of the final selected diviseme instances are time warped and blended to construct the new images of the mouth animation. Experimental results showed that the multimodal diviseme instance selection scheme of [8] outperforms the triphone unit selection algorithm in Video Rewrite. Clear, accurate, smooth mouth animations can be obtained matching well the pronunciation and intensity changes of the incoming speech.…”

“…In this paper, we expand the multi-modal diviseme unit selection based mouth animation system of [8] to a full facial animation system. The novelties of this work are: 1) to reduce the influence of head pose and face deflection, we align the face images as well their feature points to a reference face by a generalized procrustes analysis (GPA) method; 2) in order to 'align' jaw shapes to be coincident with the mouth movements in the synthesized facial animation, we identify the mapping from mouths deformations to jaws, and hence the non-rigid deformations of mouth and jaws, using the Generalized Regression Neural Networks (GRNN).…”