Recovering facial reflectance and geometry from multi-view images

Abstract

While the problem of estimating shapes and diffuse reflectances of human faces from images has been extensively studied, there is relatively less work done on recovering the specular albedo. This paper presents a lightweight solution for inferring photorealistic facial reflectance and geometry. Our system processes video streams from two views of a subject, and outputs two reflectance maps for diffuse and specular albedos, as well as a vector map of surface normals. A model-based optimization approach is used, consisting of the three stages of multi-view face model fitting, facial reflectance inference and facial geometry refinement. Our approach is based on a novel formulation built upon the 3D morphable model (3DMM) for representing 3D textured faces in conjunction with the Blinn-Phong reflection model. It has the advantage of requiring only a simple setup with two video streams, and is able to exploit the interaction between the diffuse and specular reflections across multiple views as well as time frames. As a result, the method is able to reliably recover high-fidelity facial reflectance and geometry, which facilitates various applications such as generating photorealistic facial images under new viewpoints or illumination conditions.

Fast Modelling Algorithm for Realistic Three-Dimensional Human Face for Film and Television Animation

“…Human face is a very important communication channel for human beings, and it is the carrier of human's complex expressions and language such as happiness, anger, sorrow, and happiness. However, three-dimensional (3D) modelling is a basic problem in the field of computer vision and computer graphics [1,2]. Realistic 3D face modelling has been a research hotspot in the field of computer vision and computer graphics for nearly 30 years, and it has a wide range of applications in film and television animation, human-computer interaction, video games, and communications [3,4].…”

Joint Learning of Portrait Intrinsic Decomposition and Relighting

“…For the intrinsic decomposition task, self-supervision is obtained by dense correspondence between pixels across multiple views [54], training on a sequence of multi-lit images or video streams [25,42], model-based shape reconstruction [46], or through reconstruction loss (imposing consistency between the original images and the re-rendered one from the estimated intrinsic components) while training on a mix of labeled and unlabeled datasets [37,36,16]. Here, we introduce a new self-supervised loss term that 1) reduces the need for pseudo-labels and multi-stage training [37], 2) does not require a sequence of images as input during training [25,42], 3) does not rely on strong priors posed in [54,25,46] for training in limited supervision scenarios (no labels on albedos and normals exist) where the intrinsic decomposition from single image is highly ambiguous. Furthermore, compared to [28] proposing an unsupervised intrinsic decomposition technique given multi-lit images at training, we further disentangle the lighting component from the normals, thus facilitating relighting and light transfer between a source and a target image pair.…”