MPEG‐4 facial animation in video analysis and synthesis

Abstract

Abstract-MPEG-4 supports the definition, encoding, transmission, and animation of 3-D head and body models. These features can be used for a variety of different applications ranging from low bit-rate video coding to character and avatar animation. In this paper, an entire system for the analysis of facial expressions from image sequences and their synthesis is presented. New methods for the estimation of MPEG-4 facial animation parameters as well as scene illumination are proposed. Experiments for different applications demonstrate the potential of using facial animation techniques in video analysis and synthesis. A model-based codec is presented that is able to encode head-and-shoulder video sequences at bit-rates of about 1 kbit/s. Besides the low bit-rate, many enhancements and scene modifications can be easily applied, like scene lighting changes or cloning of expressions for character animation. But also for the encoding of arbitrary sequences, 3-D knowledge can help to increase the coding efficiency. With our model-aided codec, bit-rate reductions of up to 45 % at the same quality can be achieved in comparison to standard hybrid video codecs.

Geometry-assisted image-based rendering for facial analysis and synthesis

“…In this section, we will briefly describe our original 3-D model-based coding system [12,11]. Although purely geometry-based, it is used in this work to represent global head motion (except for head turns) and jaw movements, which severely effects the silhouette if the person is viewed from a sideways directions.…”

Virtual Mirror: Real-Time Tracking of Shoes in Augmented Reality Environments

“…In total, 12 parameters are estimated using an analysis-by-synthesis technique similar to the face tracker described in [5,6]. Instead of tracking a set of distinct feature points, the entire image is exploited for robust motion estimation.…”

Optical flow based tracking and retexturing of garments

“…The problem of capturing non-rigid motion has been addressed in many fields of application including medical imaging [1], objectbased video compression [2], [3] or augmented reality [4]. It has been successfully demonstrated for human motion analysis and face tracking.…”