Vision Based Speech Animation Transferring with Underlying Anatomical Structure

Abstract

Abstract. We present a novel method to transfer speech animation recorded in low resolution videos onto realistic 3D facial models. Unsupervised learning is utilized on a speech video corpus to find underlying manifold of facial configurations. K-means clustering is applied on the low dimensional space to find key speaking-related facial shapes. With a small set of laser scanner captured 3D models related to the clustering centroid, the facial animation in 2D videos is transferred onto 3D shapes. Especially by virtue of a weak perspective projection model, the underlying mandible rotation is recovered from videos and is utilized to drive 3D skull movements. The adaption of a generic skull onto facial models is guided by a 2D image, Tissue Map. With parsimonious data requirements, our system realizes the animation transferring and gains a realistic rendering effect with the underlying anatomical structure.

Towards a Northern Sotho talking head

“…Phoneme-viseme correspondence and co-articulation remains a challenge in speech-synthesized facial animation [Deng et al 2006;Leszczynski and Skarbek 2005;Massaro et al 2006;Sifakis et al 2006], particularly for languages that are significantly different from English [King and Parent 2005;Pei and Zha 2006]. In our experiment we associated various phonemes of Northern Sotho that do not also occur in English with blends of visemes for English -the process was not scientifically sophisticated, but rather subjective, manual, time-consuming and trial-and-error driven.…”