AvatarMe: Realistically Renderable 3D Facial Reconstruction “In-the-Wild”

Lattas, Alexandros; Moschoglou, Stylianos; Geçer, Barış; Ploumpis, Stylianos; Triantafyllou, Vasileios; Ghosh, Abhijeet; Zafeiriou, Stefanos

doi:10.1109/cvpr42600.2020.00084

Cited by 137 publications

(72 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Apart from collecting such datasets, which is an expensive and tedious task, several interesting alternatives exist. For example, Generative Adversarial Networks can be used to perform image-to-image translation to obtain different views for existing datasets [27], [28], 3D models can be created from real images and the various views can be directly rendered [29], both real and simulated data can be combined using highly realistic simulations [30], [31], while knowledge distillation methods could be used to further reduce the gap between real and simulated data [32], [33].…”

Section: Discussionmentioning

confidence: 99%

Leveraging Active Perception for Improving Embedding-based Deep Face Recognition

Passalis

Tefas

2020

2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)

View full text Add to dashboard Cite

Even though recent advances in deep learning (DL) led to tremendous improvements for various computer and robotic vision tasks, existing DL approaches suffer from a significant limitation: they typically ignore that robots and cyberphysical systems are capable of interacting with the environment in order to better sense their surroundings. In this work we argue that perceiving the world through physical interaction, i.e., employing active perception, allows for both increasing the accuracy of DL models, as well as for deploying smaller and faster models. To this end, we propose an active perception-based face recognition approach, which is capable of simultaneously extracting discriminative embeddings, as well as predicting in which direction the robot must move in order to get a more discriminative view. To the best of our knowledge, we provide the first embedding-based active perception method for deep face recognition. As we experimentally demonstrate, the proposed method can indeed lead to significant improvements, increasing the face recognition accuracy up to 9%, as well as allowing for using overall smaller and faster models, reducing the number of parameters by over one order of magnitude.

show abstract

Section: Discussionmentioning

confidence: 99%

Leveraging Active Perception for Improving Embedding-based Deep Face Recognition

Passalis

Tefas

2020

2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)

View full text Add to dashboard Cite

show abstract

“…Deep learning-based inference techniques [70,67,52,27,61,65,20,5,61] were later introduced and have demonstrated significantly more robust facial digitization capabilities but they are still ineffective in capturing facial geometric and appearance detail due to the linearity and low dimensionality of the face model. Several post-processing techniques exist and use inferred linear face models to generate high-fidelity facial assets such as albedo, normal, and specular maps for relightable avatar rendering [41,16,72]. AvatarMe [41] for instance uses GANFIT [67] to generate a linear 3DMM model as input to their post processing framework.…”

Section: Related Workmentioning

confidence: 99%

“…Several post-processing techniques exist and use inferred linear face models to generate high-fidelity facial assets such as albedo, normal, and specular maps for relightable avatar rendering [41,16,72]. AvatarMe [41] for instance uses GANFIT [67] to generate a linear 3DMM model as input to their post processing framework. Our proposed method can be used as alternative input to AvatarMe, and we compare it to GANFIT later in Section 4.…”

Section: Related Workmentioning

confidence: 99%

Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement

Luo¹,

Nagano²,

Kung³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce a highly robust GAN-based framework for digitizing a normalized 3D avatar of a person from a single unconstrained photo. While the input image can be of a smiling person or taken in extreme lighting conditions, our method can reliably produce a high-quality textured model of a person's face in neutral expression and skin textures under diffuse lighting condition. Cutting-edge 3D face reconstruction methods use non-linear morphable face models combined with GAN-based decoders to capture the likeness and details of a person but fail to produce neutral head models with unshaded albedo textures which is critical for creating relightable and animation-friendly avatars for integration in virtual environments. The key challenges for existing methods to work is the lack of training and ground truth data containing normalized 3D faces. We propose a two-stage approach to address this problem. First, we adopt a highly robust normalized 3D face generator by embedding a non-linear morphable face model into a StyleGAN2 network. This allows us to generate detailed but normalized facial assets. This inference is then followed by a perceptual refinement step that uses the generated assets as regularization to cope with the limited available training samples of normalized faces. We further introduce a Normalized Face Dataset, which consists of a combination photogrammetry scans, carefully selected photographs, and generated fake people with neutral expressions in diffuse lighting conditions. While our prepared dataset contains two orders of magnitude less subjects than cutting edge GAN-based 3D facial reconstruction methods, we show that it is possible to produce high-quality normalized face models for very challenging unconstrained input images, and demonstrate superior performance to the current state-of-the-art.

show abstract

“…This was the first work to introduce 3D Morphable Model (3DMM) for facial shape reconstruction using a PCA-based linear subspace that captures shape variations in human faces. With the emergence of DL techniques, more advanced MFSR approaches have been developed, such as RSNIEF [14], UH-E2FAR [15], Ganfit [16], MMFace [17], and AvatarMe [18].…”

Section: D Fe Synthesismentioning

confidence: 99%

“…The 3D face geometry reconstruction approach from 2D facial images [13] is restricted due to the various faced challenges by the used algorithm, such as illumination conditions and diversified FEs. In addition, exploiting the 3DMM [21,22] fails to accurately depict the complicated structure of facial details [18].…”

Section: D Fe Synthesismentioning

confidence: 99%

Facial Expression Recognition Using 3D Points Aware Deep Neural Network

Trimech¹,

Maalej²,

Amara³

2021

View full text Add to dashboard Cite

Point cloud-based Deep Neural Networks (DNNs) have gained increasing attention as an insightful solution in the study field of geometric deep learning. Point set aware DNNs have proven capable of dealing with the unstructured data type and successful in 3D data applications such as 3D object classification, segmentation and recognition. On the other hand, two major challenges remain understudied when it comes to the use of point cloud-based DNNs for 3D facial expression (FE) recognition. The first challenge is the lack of large labelled 3D facial data. The second is how to obtain a point-based discriminative representation of 3D faces. To address the first issue, we suggest to enlarge the used dataset by generating synthetic 3D FEs. For the second one, we propose to apply a level-curve based sampling strategy in order to exploit crucial geometric information. The conducted experiments show promising results reaching 97.23% on the enlarged BU-3DFE dataset.

show abstract

AvatarMe: Realistically Renderable 3D Facial Reconstruction “In-the-Wild”

Cited by 137 publications

References 35 publications

Leveraging Active Perception for Improving Embedding-based Deep Face Recognition

Leveraging Active Perception for Improving Embedding-based Deep Face Recognition

Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement

Facial Expression Recognition Using 3D Points Aware Deep Neural Network

Contact Info

Product

Resources

About