Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis

Abstract

Photorealistic frontal view synthesis from a single face image has a wide range of applications in the field of face recognition. Although data-driven deep learning methods have been proposed to address this problem by seeking solutions from ample face data, this problem is still challenging because it is intrinsically ill-posed. This paper proposes a Two-Pathway Generative Adversarial Network (TP-GAN) for photorealistic frontal view synthesis by simultaneously perceiving global structures and local details. Four landmark located patch networks are proposed to attend to local textures in addition to the commonly used global encoderdecoder network. Except for the novel architecture, we make this ill-posed problem well constrained by introducing a combination of adversarial loss, symmetry loss and identity preserving loss. The combined loss function leverages both frontal face distribution and pre-trained discriminative deep face models to guide an identity preserving inference of frontal views from profiles. Different from previous deep learning methods that mainly rely on intermediate features for recognition, our method directly leverages the synthesized identity preserving image for downstream tasks like face recognition and attribution estimation. Experimental results demonstrate that our method not only presents compelling perceptual results but also outperforms state-of-theart results on large pose face recognition. * These two authors contributed equally.† Homepage

Mask-Guided Portrait Editing With Conditional GANs

“…Recently, advances in Generative Adversarial Networks (GANs) [16] have made tremendous progress in synthesizing realistic faces [1,29,25,12], like face aging [46], pose changing [44,21] and attribute modifying [4]. However, these existing approaches still suffer from some quality issues, like lack of fine details in skin, difficulty in dealing with hair and background blurring.…”

On the benefit of adversarial training for monocular depth estimation

“…This may be due to the fact that the adversarial loss punishes high-level detail, but only slowly updates low-level detail. Combining adversarial losses and pixelbased local losses has been shown to work well for a number of tasks, including novel viewpoint estimation (Huang et al, 2017;Wu et al, 2016;Galama and Mensink, 2018), predicting future frames in a video (Yin et al, 2018;Mathieu et al, 2015), and image inpainting (Pathak et al, 2016).…”

Sampling Strategies for GAN Synthetic Data

“…Especially, GANs are known to be able to generate realistic samples, while the discriminator and the generator play a "two-player minimax game". Generating new type data using GANs and augementing with real data has been investigated in recent works (Baek, Kim, and Kim 2018;Gecer et al 2018;Zhang et al 2018;Shmelkov, Schmid, and Alahari 2018;Zhao et al 2018b;Tran, Yin, and Liu 2017;Zhao et al 2018a;Huang et al 2017) and too few to mention. In this paper, we try to investigate methods and tricks to sub-sample instead of randomly augmenting the synthetic images from GAN.…”