Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors

Abstract

In this paper, we present supervision-by-registration, an unsupervised approach to improve the precision of facial landmark detectors on both images and video. Our key observation is that the detections of the same landmark in adjacent frames should be coherent with registration, i.e., optical flow. Interestingly, the coherency of optical flow is a source of supervision that does not require manual labeling, and can be leveraged during detector training. For example, we can enforce in the training loss function that a detected landmark at frame t−1 followed by optical flow tracking from frame t−1 to frame t should coincide with the location of the detection at frame t . Essentially, supervisionby-registration augments the training loss function with a registration loss, thus training the detector to have output that is not only close to the annotations in labeled images, but also consistent with registration on large amounts of unlabeled videos. End-to-end training with the registration loss is made possible by a differentiable Lucas-Kanade operation, which computes optical flow registration in the forward pass, and back-propagates gradients that encourage temporal coherency in the detector. The output of our method is a more precise image-based facial landmark detector, which can be applied to single images or video. With supervision-by-registration, we demonstrate (1) improvements in facial landmark detection on both images (300W, ALFW) and video (300VW, Youtube-Celebrities), and (2) significant reduction of jittering in video detections.

Learning Multi-human Optical Flow

“…Similar work [25,26] propagates pose results temporally using optical flow to encourage time consistency of the estimated bodies. Apart from its application in warping between frames, the structural information existing in optical flow alone has been used for pose estimation [27] or in conjunction with an image stream [28,29].…”

Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection

“…Linear regression based methods learn a function that maps the input face image to the normalized landmark coordinates [44,7]. Heatmap regression based methods produce one heatmap for each landmark, where the coordinate is the location of the highest response on this heatmap [41,11,9,30,5]. All above algorithms can be readily integrated into our framework, serving as different student detectors.…”

“…The 300-W dataset [35] annotates 68 landmarks from five facial landmark datasets, i.e., LFPW, AFW, HELEN, XM2VTS, and IBUG. Following the common settings [11,9,27], we regard all the training samples from LFPW, HE-LEN and the full set of AFW as the training set, in which there is 3148 training images. The common test subset consists of 554 test images from LFPW and HELEN.…”

“…Normalized Mean Error (NME) is usually applied to evaluate the performance for facial landmark predictions [27,34,47,9]. For the 300-W dataset, we use the inter-ocular distance to normalize mean error following the same setting as in [35,27,11,9]. For the AFLW dataset, we use the face size to normalize mean error [27].…”

DeCaFA: Deep Convolutional Cascade for Face Alignment in the Wild

“…More recently, PCD-CNN [9] uses head pose information to drive the training process. CPM+SBR [5] employs landmark registration to regularize training. SAN [4] uses adversarial networks to convert images from different styles to an aggregated style, upon which regression is performed.…”