DeepSketch2Face

Abstract

Fig. 1. Using our sketching system, an amateur user can create 3D face or caricature models with complicated shape and expression in a few minutes. Both models shown here were created in less than 10 minutes by a user without any prior drawing and modeling experiences.Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.

Tag-based annotation creates better avatars

“…Stylized avatar systems also exist. Some methods utilize sketches as the prior condition for generation 22,23 . Other methods are guided by position and landmarks, extracting human facial features used to deform textures and meshes [24][25][26][27] .…”

Deep Reconstruction of 3D Smoke Densities from Artist Sketches

“…A deep learning based sketching system has also been used for 3D face and caricature modeling [HGY17]. 2D lines representing the contours of facial features represent the input to a CNN.…”

SimpModeling: Sketching Implicit Field to Guide Mesh Modeling for 3D Animalmorphic Head Design

“…To demonstrate the effectiveness of our system, we carefully conduct user studies, from which we conclude that our proposed system exhibits better usability than existing sketch-based 3D face modeling systems [10], [11]. Our system allows amateur users to create diverse shapes with fine-grained geometric details.…”

“…Several intelligent sketchbased modeling systems have been developed to enable novice users to create visually plausible 3D models within a few minutes [7], [8], [9]. Closest to our work, DeepSketch2Face [10] presents the first deep learning-based sketching system for modeling 3D faces by mapping 2D sketches into a parametric space for face generation. However, considering the limited representa-• Z.…”

“…Accuracy. Recent learning-based sketching systems [10], [11], [14], [15], [16] allow novice users to create visually-plausible 3D models with a few strokes. However, they are not capable of designing shapes with fine-grained details.…”