Unsupervised Facial Geometry Learning for Sketch to Photo Synthesis

Abstract

Face sketch-photo synthesis is a critical application in law enforcement and digital entertainment industry where the goal is to learn the mapping between a face sketch image and its corresponding photo-realistic image. However, the limited number of paired sketch-photo training data usually prevents the current frameworks to learn a robust mapping between the geometry of sketches and their matching photo-realistic images. Consequently, in this work, we present an approach for learning to synthesize a photo-realistic image from a face sketch in an unsupervised fashion. In contrast to current unsupervised image-to-image translation techniques, our framework leverages a novel perceptual discriminator to learn the geometry of human face. Learning facial prior information empowers the network to remove the geometrical artifacts in the face sketch. We demonstrate that a simultaneous optimization of the face photo generator network, employing the proposed perceptual discriminator in combination with a texture-wise discriminator, results in a significant improvement in quality and recognition rate of the synthesized photos. We evaluate the proposed network by conducting extensive experiments on multiple baseline sketch-photo datasets.

Identity-aware CycleGAN for face photo-sketch synthesis and recognition

“…Recently, Wang et al [21] applied the multi-adversarial idea to CycleGAN to generate high-resolution images. For sketch-to-photo synthesis, Kazemi et al [22] employ an additional geometrydiscriminator to distinguish based on high-level facial features. However, these approaches have only focused on adapting the architecture of GAN for an improvement on synthesizing images.…”

“…Since we have paired data in the target datasets, unsupervised training in [8,22] is not necessary for our work. To learn the identity information of each person, high-level features of facial images are extracted from a pretrained recognition model that is fine-tuned based on the VGGFace model [26].…”

Transporting Labels via Hierarchical Optimal Transport for Semi-Supervised Learning

“…where γ is a hyperparameter that balances two terms in (9), and E(T ) = − mn ij T ij (log(T ij − 1) is the entropy of the solution matrix T . It has been shown that if T γ is the solution of the optimization (9), then ∃!u ∈ R n + , v ∈ R m + such that the solution matrix for (9) is T γ = diag(u)Kdiag(v) where, K = exp(−X/γ) [30]. The vectors u and v are updated iteratively between step 1 and 2 by using the well-known Sinkhorn algorithm as follows: step 1)u = a/Kv and step 2)v = b/K u, where/ denotes element-wise division operator [30].…”

“…Recent developments in CNNs have provided promising results for many applications in machine learning and computer vision such as facial recognition [1,2,3,4,5,6], image retrieval [4,7,8,9], image generation [10,11,10], and adversarial attack [12,13]. However, the success of CNN models requires a vast amount of well-annotated training data, which is not always feasible to perform manually [14,15].…”

From 2D Sketches to Photo-Realistic Images using Generative Adversarial Networks

“…Facial texture can be used while outputting the final image for a more accurate prediction. [2] Phillip Isola, proposed a system for image to image translation with the help of Generative Adversarial Networks to predict results that were very close to the ground truth. The system had high complexity due to a generalized approach towards image translation.…”