Avatar digitization from a single image for real-time rendering

Abstract

We present a fully automatic framework that digitizes a complete 3D head with hair from a single unconstrained image. Our system offers a practical and consumer-friendly end-to-end solution for avatar personalization in gaming and social VR applications. The reconstructed models include secondary components (eyes, teeth, tongue, and gums) and provide animation-friendly blendshapes and joint-based rigs. While the generated face is a high-quality textured mesh, we propose a versatile and efficient polygonal strips (polystrips) representation for the hair. Polystrips are suitable for an extremely wide range of hairstyles and textures and are compatible with existing game engines for real-time rendering. In addition to integrating state-of-the-art advances in facial shape modeling and appearance inference, we propose a novel single-view hair generation pipeline, based on 3D-model and texture retrieval, shape refinement, and polystrip patching optimization. The performance of our hairstyle retrieval is enhanced using a deep convolutional neural network for semantic hair attribute classification. Our generated models are visually comparable to state-of-the-art game characters designed by professional artists. For real-time settings, we demonstrate the flexibility of polystrips in handling hairstyle variations, as opposed to conventional strand-based representations. We further show the effectiveness of our approach on a large number of images taken in the wild, and how compelling avatars can be easily created by anyone.

DeepSketchHair: Deep Sketch-Based 3D Hair Modeling

“…Chai et al [28] introduce a deep convolutional network for segmenting hair regions and estimating hair growth orientation, which guides their datadriven hair modeling method. A similar idea is adopted by Hu et al [2] for hair digitization, which uses neural networks for hair classification and region segmentation. Zhou et al [7] present an encoder-decoder network architecture to generate strand features from 2D orientation input for hair growing.…”

Combining 3D Morphable Models: A Large Scale Face-And-Head Model

“…An extension of this work is proposed in [24] in which a non-linear model is constructed using convolution mesh autoencoders focusing on facial expressions, but still it lacks the statistical variation of the full cranium. Similarly, in the work of Hu and Saito [16], a full head model is created from single images mainly for real-time rendering. The work aims at creating a realistic avatar model which includes 3D hair estimation.…”

Hair-GAN: Recovering 3D hair structure from a single image using generative adversarial networks

“…Chai et al [1] present a fully automated hair modeling method by replacing user interactions with deep convolutional neural networks to hair segmentation and hair growth direction estimation. Hu et al [10] introduce deep learning based hair attribute classifier to improve the candidate retrieval performance of the data-driven method. In order to get an end-to-end learning from 2D image knowledge to 3D hair representation, Zhou et al [11] use encoder-decoder architecture to generate hair strands represented as sequences of 3D points for 2D orientation fields as input.…”