How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks)

Abstract

This paper investigates how far a very deep neural network is from attaining close to saturating performance on existing 2D and 3D face alignment datasets. To this end, we make the following 5 contributions: (a) we construct, for the first time, a very strong baseline by combining a state-of-the-art architecture for landmark localization with a state-of-the-art residual block, train it on a very large yet synthetically expanded 2D facial landmark dataset and finally evaluate it on all other 2D facial landmark datasets. (b) We create a guided by 2D landmarks network which converts 2D landmark annotations to 3D and unifies all existing datasets, leading to the creation of LS3D-W, the largest and most challenging 3D facial landmark dataset to date (~230,000 images). (c) Following that, we train a neural network for 3D face alignment and evaluate it on the newly introduced LS3D-W. (d) We further look into the effect of all "traditional" factors affecting face alignment performance like large pose, initialization and resolution, and introduce a "new" one, namely the size of the network. (e) We show that both 2D and 3D face alignment networks achieve performance of remarkable accuracy which is probably close to saturating the datasets used. Training and testing code as well as the dataset can be downloaded from https: //www.adrianbulat.com/face-alignment/ arXiv:1703.07332v3 [cs.CV]

The 3D Menpo Facial Landmark Tracking Challenge

“…a model of shape variation across different individuals, assuming that all shapes are under neutral expression. For this, we adopt our LSFM models [2], which consist the largest models of 3D Morphable Modelling (3DMM) of facial identity built from approximately 10,000 scans of different individuals 4 . The dataset that LSFM models are trained on includes rich demographic information about each subject, allowing the construction of not only a global 3DMM model but also bespoke models tailored for specific age, gender or ethnicity groups.…”

A Cross‐Dimension Annotations Method for 3D Structural Facial Landmark Extraction

“…[JL16] proposed a cascaded CNN regressor and achieved state‐of‐the‐art performance on two challenging face databases with large poses. [BT17] proposed a direct conversion from 2D landmarks to 3DA‐2D using deep CNNs. They also published a large image data set LS3D‐W annotated with 3DA‐2D landmarks—the largest and most challenging 3D facial landmark data set to date (about 230 000 images).…”

“…In [BT17], for the first time, 2D landmarks are converted to 3DA‐2D landmarks using a deep neural network. This approach won the first place in the First 3D Face Alignment in the Wild (3DFAW) Challenge .…”

“…A large data set annotated by 3DA‐2D, i.e. LS3D‐W, was published with [BT17]. The core of [BT17] is a Cross‐Dimension Annotations Conversion (CDAC) that transforms 2D landmarks to 3DA‐2D.…”

A Coarse-to-Fine Approach for 3D Facial Landmarking by Using Deep Feature Fusion

“…In Zhu [11], Jourabloo [12] and Kumar [32], they all adopt a 3D solution in a novel alignment framework, which shows that the character of 3D data can help to conquer the limitation of arbitrary pose and other challenges. In Bulat [33], they created a large dataset and estimated 2D and 3D landmarks by adopting hourglass networks. However, all of these methods obtain corresponding 3D shape by adopting 3DMM or 2D texture images that is also sensitive to the changeable lighting conditions.…”