InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

Abstract

Although Generative Adversarial Networks (GANs) have made significant progress in face synthesis, there lacks enough understanding of what GANs have learned in the latent representation to map a randomly sampled code to a photo-realistic face image. In this work, we propose a framework, called InterFaceGAN, to interpret the disentangled face representation learned by the state-of-the-art GAN models and thoroughly analyze the properties of the facial semantics in the latent space. We first find that GANs actually learn various semantics in some linear subspaces of the latent space when being trained to synthesize high-quality faces. After identifying the subspaces of the corresponding latent semantics, we are able to realistically manipulate the facial attributes occurring in the synthesized images without retraining the model. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection, resulting in more precise control of the attribute manipulation. Besides manipulating gender, age, expression, and the presence of eyeglasses, we can even alter the face pose as well as fix the artifacts accidentally generated by GANs. Furthermore, we perform in-depth face identity analysis and layer-wise analysis to quantitatively evaluate the editing results. Finally, we apply our approach to real face editing by involving GAN inversion approaches as well as explicitly training additional feed-forward models based on the synthetic data established by InterFaceGAN. Extensive experimental results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable face representation.

Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation

“…It suggests that our method can achieve fine-grained controls on local semantic regions of generated images. Note that, our proposed method can perform various local attribute editing tasks, which is much more than previous methods [10,25,26,33]. Then we further visualize the control units for several attribute manipulations in Figure 4.…”

“…We validated our approach on a variety of local attribute editing tasks. We apply InterfaceGAN [25] to find the initial direction vectors to be cropped corresponding to those attributes that have annotated by [20]. For other attributes that cannot train an available classifier, we directly use the difference between the modulation styles of a few annotated positive and negative samples as the direction vectors.…”

“…The approaches of semantic image editing with StyleGAN can be roughly divided into two groups, i.e., supervised approaches and unsupervised approaches. The supervised approaches [3,9,13,14,19,25] introduce pretrained classifiers to find the directions that alter the output of the classifiers. For example, InterfaceGAN [1] trains linear support vector machines (SVMs) using the attribute annotations labeled by the off-the-shelf classifiers and finds hyperplanes in the latent space serving as the separation boundary.…”

“…Previous works [10,25,26] can identify a semantically meaningful direction vector in the W or Z space that can manipulates target attribute of 𝑟 . We first transformed these direction vectors found by other methods into the S space.…”

“…The key problem in attribute manipulation with StyleGAN is how to obtain a fine-grained controls, i.e., semantic and spatial disentangled controls. A well-trained StyleGAN inherently encodes numerous semantic information in the latent space and hidden features [5,22,24,25,30,34,35]. GAN Dissection [5] found the causal feature maps are specialized to synthesize specific visual arXiv:2111.13010v1 [cs.CV] 25 Nov 2021 concepts in generated images.…”