Automatic 3D Facial Expression Analysis in Videos

Abstract

Abstract. We introduce a novel framework for automatic 3D facial expression analysis in videos. Preliminary results demonstrate editing facial expression with facial expression recognition. We first build a 3D expression database to learn the expression space of a human face. The real-time 3D video data were captured by a camera/projector scanning system. From this database, we extract the geometry deformation independent of pose and illumination changes. All possible facial deformations of an individual make a nonlinear manifold embedded in a high dimensional space. To combine the manifolds of different subjects that vary significantly and are usually hard to align, we transfer the facial deformations in all training videos to one standard model. Lipschitz embedding embeds the normalized deformation of the standard model in a low dimensional generalized manifold. We learn a probabilistic expression model on the generalized manifold. To edit a facial expression of a new subject in 3D videos, the system searches over this generalized manifold for optimal replacement with the 'target' expression, which will be blended with the deformation in the previous frames to synthesize images of the new expression with the current head pose. Experimental results show that our method works effectively.

Recognition of 3D facial expression dynamics

“…3D data was used for analysis of expression dynamics in [18], in which a small 3D database was created which could be used for the analysis. Feature points were tracked in order to capture the deformation of the 3D mesh during the expression.…”

“…However these methods are also susceptible to the problems of illumination and pose inherent to all 2D methods. For this reason, more recently several methods have been proposed which Image and Vision Computing 30 (2012) [762][763][764][765][766][767][768][769][770][771][772][773] use 3D facial geometry data for facial expression recognition, either for static analysis [11][12][13][14][15][16], to encode the temporal information [16,17], or to model the temporal dynamics of facial expressions in 3D image sequences [18,19].…”

Facial Expression Recognition Based on 3D Dynamic Range Model Sequences

“…However, the 3D models that have been used are all static. The most recent technological advances in 3D imaging allow for real-time 3D facial shape acquisition [18,19] and analysis [20]. Such 3D sequential data captures the dynamics of time-varying facial surfaces, thus allowing us to use 3D dynamic surface features or 3D motion units (rather than 2D motion units) to scrutinize facial behaviors at a detailed level.…”

“…Wang et al [18] have successfully developed a hierarchical framework for tracking high-density 3D facial sequences. The recent work in [20] utilized dynamic 3D models of six subjects for facial analysis and editing based on the generalized facial manifold of a standard model.…”

Automatic Facial Pose Determination of 3D Range Data for Face Model and Expression Identification

“…Unfortunately, 2D data is unsatisfactory because it is inherently unable to handle faces with large head rotation, subtle skin movement, or lighting change with varying postures. With the recent advance of 3D imaging systems [23,11], research on face and facial expression recognition using 3D data has been intensified [8, 9, 10, 16, 19, and 22]. However, almost all existing 3D-based recognition systems are based on static 3D facial data.…”