Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach

Automatic 3D face reconstruction from single images or video

“…This is in contrast to previous work in model-based shape reconstruction from monocular video, which involved an analysis of multiple frames, such as model-driven bundleadjustment [13], least-squares reconstruction from multiple successive frames [11], structure-from-motion with subsequent refinement by a deformable face model [8], nonrigid structure-from-motion with intrinsic model constraints [5] and feature tracking and factorization of the tracking matrix for non-rigid shape estimation [6]. Zhang et al [28] presented an algorithm that involves tracking, model fitting and multiple-view bundle adjustment. Many of these algorithms require manual interaction such as a number of mouse clicks.…”

Model-based human shape reconstruction from multiple views

“…This approach has been used widely to derive animated models of the human face, see for example [5,19]. Whole-body human shape reconstruction has been presented previously to recover either static shape and articulated motion [4,20] or dynamic shape [21] from image silhouettes and from multiple shape cues [9].…”

“…These techniques have no prior scene structure to construct a consistent representation that can be instrumented for animation and synthesis of new content. Model-based scene reconstruction is a well established approach [3][4][5] in which a prior scene model is adapted to fit available shape data and so provides a consistent structure that can be used for synthesis. Previous model-fitting techniques have constrained the freedom in model deformation for robust reconstruction by restricting the space of feasible shape using a coarse input model or by restricting models to closed surfaces.…”

Learning 3D appearance models from video

“…Chowdhury and Chellappa [7] construct a 3D model by inferring depth from flow. In a similar approach but using feature correspondences and performing bundle adjustment, Zhang et al [21] construct 3D models from a video in which the face rotates 180 degrees from profile to profile. Pighin et al [16] model and animate 3D Face Models using SFM in multi-view images and solving the correspondence by hand.…”

“…Among various approaches to modeling 3D faces from video, two of the most popular and commonly used are based on appearance models (AM) [2,4,8,9,17] and rigid/nonrigid structure from motion (SFM) [5, Figure 1: Generative 3D Facial Appearance Model with Structure and Appearance. 7,16,21]. While each has been studied extensively, both approaches suffer from several drawbacks.…”