Learning Detailed Face Reconstruction from a Single Image

Abstract

Reconstructing the detailed geometric structure of a face from a given image is a key to many computer vision and graphics applications, such as motion capture and reenactment. The reconstruction task is challenging as human faces vary extensively when considering expressions, poses, textures, and intrinsic geometries. While many approaches tackle this complexity by using additional data to reconstruct the face of a single subject, extracting facial surface from a single image remains a difficult problem. As a result, single-image based methods can usually provide only a rough estimate of the facial geometry. In contrast, we propose to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image. For this purpose, we introduce an end-to-end CNN framework which derives the shape in a coarse-to-fine fashion. The proposed architecture is composed of two main blocks, a network that recovers the coarse facial geometry (CoarseNet), followed by a CNN that refines the facial features of that geometry (FineNet). The proposed networks are connected by a novel layer which renders a depth image given a mesh in 3D. Unlike object recognition and detection problems, there are no suitable datasets for training CNNs to perform face geometry reconstruction. Therefore, our training regime begins with a supervised phase, based on synthetic images, followed by an unsupervised phase that uses only unconstrained facial images. The accuracy and robustness of the proposed model is demonstrated by both qualitative and quantitative evaluation tests.

FACSIMILE: Fast and Accurate Scans From an Image in Less Than a Second

“…The solution to this problem can be traced all the way back to Shape From Shading (SFS) literature by Horn [18], in which surface normals play a critical role in defining the relationship between a surface and its appearance. Work focused in the reconstruction of the face region [35] has shown that a loss on depth can benefit from an additional loss on normals. We go beyond this insight showing that a loss just on normals can be sufficient to reconstruct a highquality depth map up to scale, and that this applies for an articulated, far from spherical object.…”

State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications

“…Besides In a follow-up work [RSOK17], surface refinement is also phrased as a learning task. The network is trained end-to-end for regressing coarse shape and a fine-scale detail layer.…”

Hyperparameter-Free Losses for Model-Based Monocular Reconstruction

“…where [·, ·] is the concatenation operator. Note that the only difference with respect to [24] is that we are assuming a pinhole camera model instead of a weak perspective model. The L Coarse does not balance the errors produced byq andt.…”