Unconstrained realtime facial performance capture

Abstract

We introduce a realtime facial tracking system specifically designed for performance capture in unconstrained settings using a consumer-level RGB-D sensor. Our framework provides uninterrupted 3D facial tracking, even in the presence of extreme occlusions such as those caused by hair, hand-to-face gestures, and wearable accessories. Anyone's face can be instantly tracked and the users can be switched without an extra calibration step. During tracking, we explicitly segment face regions from any occluding parts by detecting outliers in the shape and appearance input using an exponentially smoothed and user-adaptive tracking model as prior. Our face segmentation combines depth and RGB input data and is also robust against illumination changes. To enable continuous and reliable facial feature tracking in the color channels, we synthesize plausible face textures in the occluded regions. Our tracking model is personalized on-the-fly by progressively refining the user's identity, expressions, and texture with reliable samples and temporal filtering. We demonstrate robust and high-fidelity facial tracking on a wide range of subjects with highly incomplete and largely occluded data. Our system works in everyday environments and is fully unobtrusive to the user, impacting consumer AR applications and surveillance.

Reconstruction of Personalized 3D Face Rigs from Monocular Video

“…Generic blendshape models are used by some face tracking methods from monocular RGB video [Garrido et al 2013] or RGB-D video [Weise et al 2011] but need to be deformed into a static face scan or a set of scanned static expressions of an actor prior to tracking. Such generic blendshape adaptation fails to capture person-specific expression details, which is why some recent approaches estimate identity and blendshape parameters from captured face animations, and also person-specific correctives on top of this generic face model [Bouaziz et al 2013;Li et al 2013;Hsieh et al 2015]. However, all these approaches require RGB-D camera input.…”

State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications

“…They use the depth and color information of an RGB-D camera to robustly segment the face region. Image taken from [HMYL15]. Figure 8: An RGB face tracker that is robust to occlusion has been proposed by [SLL16].…”

“…In general, they are based on a segmentation mask that disables the data fitting terms in occluded regions. Hsieh et al [HMYL15] use depth and color information of an RGB-D camera to robustly segment the visible face region (see Fig. 7).…”

Realtime Dynamic 3D Facial Reconstruction for Monocular Video In-the-Wild

“…An adaptive scheme was proposed to capture more detail with point-to-point deformation on top of blendshapes in [22]. To explicitly deal with outliers caused by occlusions, a method was proposed to segment the face and complete the occluded parts based on the blendshape in [19], which was later extended to RGB input in [28]. Binocular stereo system, on the other hand, can provide higher resolution and work in outdoor environments directly under sunlight, but are more prone to suffer from lighting variation.…”