3D vs. 2D: On the Importance of Registration for Hallucinating Faces Under Unconstrained Poses

Abstract

Face Hallucination (FH) differs from generic singleimage super-resolution (SR) algorithms in its specific domain of application. By exploiting the common structures of human faces, magnification of lower resolution images can be achieved. Despite the growing interest in recent years, considerably less attention is paid to a crucial step in FH-registration of facial images. In this work, registration techniques employed in the literature are first summarized and the importance of using well-aligned training and test images is demonstrated. A novel method to inversely map the high-resolution (HR) 3D training texture to the low-resolution (LR) 2D test image in arbitrary poses is then presented, which prevents information loss in LR images and is thus beneficial to SR. The effectiveness of our 3D approach is evaluated on the Multi-PIE [1] and the PUT [2] face databases. Superior qualitative and quantitative FH results to the state-ofthe-art methods in all tested poses prove the necessity of accurate registration in FH. The merit of 3D FH in generating superresolved frontal faces is also verified, revealing 30% improvement in face recognition over the 2D approach under 30°of yaw rotation on the Multi-PIE [1] dataset.

Robust 3D Patch-Based Face Hallucination

“…Discussion Although existing 2.5D algorithms [29,33] have so far demonstrated impressive competence in both FH and FR on synthetic LR data, we observe several areas of possible improvements: (1) 3D shape reconstruction is performed either with a standard 3DMM fitting algorithm [9] in [29] or solely based on a few automatically detected facial landmarks in [33], of which the quality is susceptible to image resolution [18,19]. (2) While rendering novel views of the super-resolved face, the real texture in the self-occluded region remains intractable [33]. These unfavorable aspects can be addressed elegantly with our proposed 3D approach.…”

“…3D-assisted FH has shown its strong potential for the first time in [29], in which Mortazavian et al warped LR images onto a predefined canonical grid by fitting a 3D model [9] before performing [3]. Qu et al [33] argued the negative impact when interpolating LR input and presented a "resolution-aware" approach akin to [13] in conjunction with patch-based FH [28]. The impact of various 2D and 3D alignment techniques was discussed, too.…”

“…Subsequently, 2D training images are rendered using HR 3D textures either on a flattened grid [29] or directly on the LR input frame [33] to run 2D FH.…”

“…The prerequisite for leveraging the prior knowledge in FH is the well registered high-resolution (HR) training data and low-resolution (LR) query image, where 3D fitting is theoretically the best solution to compensate for complex global motion and local deformation of human faces [33]. Nevertheless, fitting 3D models to 2D images is a challenging task, and the LR input could make things even worse for lack of high-frequency facial details.…”

“…Nevertheless, fitting 3D models to 2D images is a challenging task, and the LR input could make things even worse for lack of high-frequency facial details. Moreover, existing 3D approaches [29,33] utilize 3D models solely as an advanced alignment tool and render the 3D training textures as 2D images for conventional 2D FH in the sequel, which, strictly speaking, can only be regarded as 2.5D methods.…”