Accurate visual speech synthesis based on diviseme unit selection and concatenation

Jiang, Dongmei; Ravyse, Ilse; Sahli, Hichem; Zhang, Yanning

doi:10.1109/mmsp.2008.4665203

Cited by 4 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is an improvement of our previous work [27] in which the values are used directly in the accumulative cost δ j n . The merit of using logarithm functions is to compress the dynamic ranges of the values, so that the weights α, β and γ can be more easily adjusted to obtain better mouth animation.…”

Section: Diviseme Instance Selection Algorithmmentioning

confidence: 95%

“…We found that for the instance selection algorithm without logarithm function as in [27], since the ranges of S ij n , B(RV j n , V n ) and C(RV j n , V n ) vary dramatically, the influences of the weights (α, β, γ ) on the mouth trajectories are not stable, i.e. it is very difficult to find the optimal setups of the weights.…”

Section: Sensitivity To the Weights α β And γmentioning

confidence: 99%

See 1 more Smart Citation

Speech driven realistic mouth animation based on multi-modal unit selection

Jiang

Ravyse

Sahli

et al. 2008

J Multimodal User Interfaces

Self Cite

View full text Add to dashboard Cite

This paper presents a novel audio visual diviseme (viseme pair) instance selection and concatenation method for speech driven photo realistic mouth animation. Firstly, an audio visual diviseme database is built consisting of the audio feature sequences, intensity sequences and visual feature sequences of the instances. In the Viterbi based diviseme instance selection, we set the accumulative cost as the weighted sum of three items: 1) logarithm of concatenation smoothness of the synthesized mouth trajectory; 2) logarithm of the pronunciation distance; 3) logarithm of the audio intensity distance between the candidate diviseme instance and the target diviseme segment in the incoming speech. The selected diviseme instances are time warped and blended to construct the mouth animation. Objective and subjective evaluations on the synthesized mouth animations prove that the multimodal diviseme instance selection algorithm proposed in this paper outperforms the triphone unit selection algorithm in Video Rewrite. Clear, accurate, smooth mouth animations can be obtained matching well with the pronunciation and intensity changes in the incoming speech. Moreover, with the logarithm function in the accumulative cost, it is easy to set the weights to obtain optimal mouth animations.

show abstract

Section: Diviseme Instance Selection Algorithmmentioning

confidence: 95%

Section: Sensitivity To the Weights α β And γmentioning

confidence: 99%

Speech driven realistic mouth animation based on multi-modal unit selection

Jiang

Ravyse

Sahli

et al. 2008

J Multimodal User Interfaces

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recently, a great deal of scientific approaches have been presented on visual speech synthesis [1][2][3][4] corresponding to novel text or prerecorded acoustic speech input. However, most of the viseme models are static.…”

Section: Introductionmentioning

confidence: 99%

Audio-Visual Speech Synthesis Based on Chinese Visual Triphone

Zhao

Chen

Shen

et al. 2009

2009 2nd International Congress on Image and Signal Processing

View full text Add to dashboard Cite

A new audio-visual speech synthesis approach is proposed based on Chinese visual triphone. Chinese visual triphone model is constructed using a new clustering method combining artificial immune system and FCM. In the analysis stage, with the training phonetic transcription, visual triphone segments are selected from video sequence, and corresponding lip feature vectors are extracted. In the synthesis stage, viterbi search algorithm is used to select the best visual triphone segments by finding out a path which produces the minimum cost. According to the concatenation principles, mouth animation is generated and stitched into background video. Experimental results show that the synthesized video is natural-looking and satisfactory.

show abstract