3DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translation

Abstract

Over the past few years, Generative Adversarial Networks (GANs) have garnered increased interest among researchers in Computer Vision, with applications including, but not limited to, image generation, translation, imputation, and super-resolution. Nevertheless, no GAN-based method has been proposed in the literature that can successfully represent, generate or translate 3D facial shapes (meshes). This can be primarily attributed to two facts, namely that (a) publicly available 3D face databases are scarce as well as limited in terms of sample size and variability (e.g., few subjects, little diversity in race and gender), and (b) mesh convolutions for deep networks present several challenges that are not entirely tackled in the literature, leading to operator approximations and model instability, often failing to preserve high-frequency components of the distribution. As a result, linear methods such as Principal Component Analysis (PCA) have been mainly utilized towards 3D shape analysis, despite being unable to capture non-linearities and high frequency details of the 3D face—such as eyelid and lip variations. In this work, we present 3DFaceGAN, the first GAN tailored towards modeling the distribution of 3D facial surfaces, while retaining the high frequency details of 3D face shapes. We conduct an extensive series of both qualitative and quantitative experiments, where the merits of 3DFaceGAN are clearly demonstrated against other, state-of-the-art methods in tasks such as 3D shape representation, generation, and translation.

Study on Automatic 3D Facial Caricaturization: From Rules to Deep Learning

“…Methods relevant to this paper are auto-encoder structures such as used by Ranjan et al (2017) and Gong et al (2019), that showcase the efficiency of recent 3D convolutional operators at capturing the distribution of 3D facial meshes. Several approaches resort to mapping 3D faces to a 2D domain, and using 2D convolution operators (Moschoglou et al, 2020). Projecting a 3D surface to a 2D plane for 2D convolutions requires locally deforming distances, which translates to higher computing and memory costs compared to recent 3D convolution approaches, and some high-frequency information loss (Gong et al, 2019).…”

3D Face Reconstruction in Deep Learning Era: A Survey

“…Besides this, the reconstruction is affected by the varying light conditions. Moschoglou et al [ 47 ] implemented an autoencoder such as 3DFaceGAN for modelling 3D facial surface distribution. Reconstruction loss and adversarial loss were used for generator and discriminator.…”

Towards a Complete 3D Morphable Model of the Human Head

“…We used 90% of the AgeDB data [51] for the training process and the rest for testing. We carried out the training utilizing a simple encoder architecture, similar to the one described in [52]. The only modification was with respect to the last layer, where the output dimension was changed to seven, to be in accordance with the total number of eye colors.…”