Robust facial landmark detection and tracking across poses and expressions for in-the-wild monocular video

Abstract

We present a novel approach for automatically detecting and tracking facial landmarks across poses and expressions from in-the-wild monocular video data, e.g., YouTube videos and smartphone recordings. Our method does not require any calibration or manual adjustment for new individual input videos or actors. Firstly, we propose a method of robust 2D facial landmark detection across poses, by combining shape-face canonical-correlation analysis with a global supervised descent method. Since 2D regression-based methods are sensitive to unstable initialization, and the temporal and spatial coherence of videos is ignored, we utilize a coarse-todense 3D facial expression reconstruction method to refine the 2D landmarks. On one side, we employ an in-the-wild method to extract the coarse reconstruction result and its corresponding texture using the detected sparse facial landmarks, followed by robust pose, expression, and identity estimation. On the other side, to obtain dense reconstruction results, we give a face tracking flow method that corrects coarse reconstruction results and tracks weakly textured areas; this is used to iteratively update the coarse face model. Finally, a dense reconstruction result is estimated after it converges. Extensive experiments on a variety of video sequences recorded by ourselves or downloaded from YouTube show the results of facial landmark detection and tracking under various lighting conditions, for various head poses and facial expressions. The overall performance and a comparison Manuscript received: 2016-09-04; accepted: 2016-12-20 with state-of-art methods demonstrate the robustness and effectiveness of our method.

Supervised coordinate descent method with a 3D bilinear model for face alignment and tracking

“…2D model‐based methods include popular AAM‐based methods: AAM‐1, and AAM‐2 . A recent 3D robust method is also compared. These methods are also trained on the same three databases.…”

“…Usually, given L labeled 2D landmarks and corresponding predefined 3D landmarks on its 3D face mesh, the pose matrix { R , t }, identity coefficients u , expression coefficients e , and displacements D = { d k } can be solved by minimizing the Huber loss function applied to re‐projected error between 2D landmarks and 3D landmarks: where P ={ R , t , u , e , D } and d k is computed on the basis of the definition above: In the case that 2D landmarks has been detected, a nonlinear trust region optimization method like a sparse variant of the Levenberg–Marquardt algorithm can be used to solve pose and bilinear parameters, as Shuang et al do. It is obvious that a reliable landmark detector is necessary, but popular detectors are usually 2D‐based and cannot capture large variations of pose and expression out of plane.…”

“…Yang et al trained linear PCA model on a public 3D face database offline, and located 2D landmarks by a variant of ASM and solved coefficients of 3D PCA components based on perspective projection when tracking online. Similar to this work, Shuang et al used a popular 2D landmark detector to localize the facial landmarks and solved the pose, expression coefficients and identity coefficients based on perspective projection and a pre‐trained 3D bilinear model. Cao et al proposed a 3D cascaded regression‐based method for user‐specific face tracking and animation.…”

Realtime Dynamic 3D Facial Reconstruction for Monocular Video In-the-Wild

“…In our experiments the landmark detector in [21] achieved the best trade-off between efficiency and accuracy, but for applications where real-time is not a priority the landmark detector could be swapped by more robust ones such as in [40,41]. To reduce redundancy only representative landmarks are chosen as described in [24] as well. The landmarks in frame i is denoted as S i , and the 3D parametric model from [6] is represented as:…”

Fast SRC using quadratic optimisation in downsized coefficient solution subspace

“…Song et al [17] proposed a half-face dictionary integration algorithm for representation-based classification, the strength of this method is that it is able to successfully construct the dual-column (row) half-face training matrix, while quantifying the integrated learning atoms that exert influence on signal reconstruction. The use of virtual face images [18,19] has also been proven beneficial to a number of face analysis tasks such as face recognition [20,21] and facial landmark detection [22,23,24,25,26]. Facial symmetry property has also been widely used to quickly locate the candidate samples in face detection, alignment and classification [27,28,29].…”