Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression

Abstract

Figure 1: A few results from our VRN -Guided method, on a full range of pose, including large expressions. Abstract 3D face reconstruction is a fundamental Computer Vision problem of extraordinary difficulty. Current systems often assume the availability of multiple facial images (sometimes from the same subject) as input, and must address a number of methodological challenges such as establishing dense correspondences across large facial poses, expressions, and non-uniform illumination. In general these methods require complex and inefficient pipelines for model building and fitting. In this work, we propose to address many of these limitations by training a Convolutional Neural Network (CNN) on an appropriate dataset consisting of 2D images and 3D facial models or scans. Our CNN works with just a single 2D facial image, does not require accurate alignment nor establishes dense correspondence between images, works for arbitrary facial poses and expressions, and can be used to reconstruct the whole 3D facial geometry (including the non-visible parts of the face) bypassing the construction (during training) and fitting (during testing) of a 3D Morphable Model. We achieve this via a simple CNN architecture that performs direct regression of a volumetric representation of the 3D facial geometry from a single 2D image. We also demonstrate how the related task of facial landmark localization can be incorporated into the proposed framework and help improve reconstruction quality, especially for the cases of large poses and facial expressions. Code and models will be made available at

Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision

“…Learning with 3D supervision: Deep learning methods are quickly replacing the optimization-based approaches [35,39,19,16]. For example, Sela et al [27] use a synthetic dataset to generate an image-to-depth mapping and a pixel-to-vertex mapping, which are combined to generate the face mesh.…”

Efficient inverse graphics in biological face processing

“…The first stage segments and normalizes the input image to compute the attended face image, i.e., the most probable value for the ideal image I * given the observed image O, by maximizing Pr(I|O) using a DCNN module trained for face volume segmentation 38 and adapted to compute the face region given images of faces with background clutter (f 1 in Fig. 1c).…”

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

“…Other representations such as volumetric grids, which do not suffer from this problem, have been also explored in the context of 3D face reconstruction. Jackson et al [122], for example, propose a Volumetric Regression Network (VRN). The framework takes as input the 2D images and their corresponding 3D binary volume instead of a 3DMM.…”