Structured Semi-supervised Forest for Facial Landmarks Localization with Face Mask Reasoning

Abstract

Motivation. Despite the great success of recent facial landmarks localization approaches, the presence of occlusions significantly degrades the performance of the systems [2,5]. Though occlusion occur frequently in realistic scenarios (e.g. the use of scarf or sunglasses, hands or hair on the face), very few works have addressed this problem explicitly due to the high diversity of occlusion in real world. While [4] tried to model a few synthetic occlusion patterns, the recent method of [1] dealt with the occlusion problem in more realistic sceneries. Both of them only focused on modelling the occlusion in an unstructured way, i.e. treating the visibility of each landmark independently. However in realistic conditions, the occlusion patterns (or called occluders) often occupy a continuous region instead of an individual pixel location, as depicted in Fig 2. Thereby the whole occluded region will consistently affect the landmarks localization.Contribution. This work attempts to address the face mask reasoning and facial landmarks localization in an unified Structured Decision Forests framework. We first have built a rich face image dataset with face mask annotation. The dataset was built as an extension of the recent datasets: Caltech Occluded Faces in the Wild (COFW), Labeled Face Parts in the Wild (LFPW) and Labeled Face in the Wild (LFW). We manually annotate a portion of images in these datasets with face masks. The face mask indicates whether or not each pixel belongs to the face. Then we incorporate such additional information of dense pixel labelling into training the Structured Classification-Regression Decision Forest. The classification nodes aim at decreasing the variance of the pixel labels of the patches by using our proposed structured criterion while the regression nodes aim at decreasing the variance of the displacements between the patches and the facial landmarks. The proposed framework allows us to predict the face mask and facial landmarks locations jointly. The proposed framework with following properties. First, semi-supervised, it uses training images from the above described augmented dataset, only a portion of which are with face masks. Second, structured, it has a novel structured criterion for split function selection for the pixel labelling (face mask reasoning) problem. Third, joint classification-regression, it predicts face mask label for each pixel (classification) and the landmark locations (regression) at the same time, and more importantly it uses the face mask reasoning results Figure 1: The framework of proposed method. We use face images with annotation of facial landmarks and face masks for training. By randomly switching the information gain function at the internal nodes, the decision trees are optimized with respect to both the offsets to landmarks (regression) and to the local structured label configuration (classification). The forest model is able to predict the face mask and landmark locations jointly. We exploit the face mask prediction to further improve the landmar...

300 Faces In-The-Wild Challenge: database and results

“…Various methods have been proposed in the literature for the task of landmark localization under semi-supervised or weakly-supervised settings [33,34,35,36]. However, there are two major limitations of these methods.…”

“…Firstly, most existing methodologies require additional information regarding the input images. Specifically, [33] employs the corresponding facial mask for each of the training images. The purpose of these masks is to indicate which pixels belong to the facial area and the only way to produce them is by manually annotating each image.…”

Random Subspace Supervised Descent Method for Regression Problems in Computer Vision

“…Comparison to state of the art: Face alignment is a very active research topic and many techniques have been proposed [5], [10], [11], [13]. Due to the limited space, we only compare the methods that are related to SDM, including the public available code of SDM (SDM-A), our implementation of SDM (SDM-B), the Incremental Face Alignment (IFA) model in [2] and CFAN [12].…”

Learning Dense Facial Correspondences in Unconstrained Images

“…While such purely 2D methods have shown impressive results especially for frontal and non-occluded faces, modeling of occlusions has been mostly avoided. More recent approaches [11,29,21,61,49] have attacked this problem by introducing occlusion variation in the training data. Handling large pose variations under difficult illumination conditions, however, still remains challenging for 2D methods.…”