Face identity disentanglement via latent space mapping

Abstract

Learning disentangled representations of data is a fundamental problem in artificial intelligence. Specifically, disentangled latent representations allow generative models to control and compose the disentangled factors in the synthesis process. Current methods, however, require extensive supervision and training, or instead, noticeably compromise quality. In this paper, we present a method that learns how to represent data in a disentangled way, with minimal supervision, manifested solely using available pre-trained networks. Our key insight is to decouple the processes of disentanglement and synthesis, by employing a leading pre-trained unconditional image generator, such as StyleGAN. By learning to map into its latent space, we leverage both its state-of-the-art quality, and its rich and expressive latent space, without the burden of training it. We demonstrate our approach on the complex and high dimensional domain of human heads. We evaluate our method qualitatively and quantitatively, and exhibit its success with de-identification operations and with temporal identity coherency in image sequences. Through extensive experimentation, we show that our method successfully disentangles identity from other facial attributes, surpassing existing methods, even though they require more training and supervision.

ShapeEditor: A StyleGAN Encoder for Stable and High Fidelity Face Swapping

“…However, it cannot accurately integrate identity and attribute information because of its simple encoder structure and the constraint of W potential space. Therefore, although it can generate high-quality images (Figure 3, column 6), it is not as good as the proposed method for fusing semantic information, which is reflected in the (Nirkin et al, 2019), FaceShifter (Li et al, 2019;Nitzan et al, 2020) on the CelebAMask-HQ (Lee et al, 2020) test dataset.…”

“…We compare the proposed method with FSGAN (Nirkin et al, 2019), FaceShifter (Li et al, 2019;Nitzan et al, 2020) on the CelebAMask-HQ (Lee et al, 2020) test dataset. Figure 3 shows, as expected because the proposed method is based on a pretrained StyleGAN (Karras et al, 2019) with high-quality face-generation capabilities, that all the generation results (Figure 3, column 6) are stable and clear enough that there are no errors such as artifacts and abnormal illumination.…”

“…This task requires more complex semantic manipulation, and the current controllers are not competent. Nitzan et al (2020) did closely related work to control expression through latent space mapping. However, working in the W space led to the failure of embedding wild images into potential space.…”

“…Our key difference is that we select W+ potential space as the mapping space, which is the premise of realizing the complex semantic operation of face swapping. In addition, in order to recover the attribute information more finely, we use multi-level feature mapping instead of a single output as attribute code like Nitzan et al (2020) did. The ablation study proves that our pertinent designs make a significant contribution to better semantic manipulation.…”

“…(Abdal et al, 2019) proved that the concatenation of 18 different 512dimensional vectors is the easiest way to embed an image and obtain a reasonable result. On this basis, various works (Gu et al, 2020;Härkönen et al, 2020;Richardson et al, 2020;Zhu et al, 2020) explore in detail the StyleGAN potential vector space: some Tewari et al, 2020) find a linear direction to control the change of a single facial attribute, some (Nitzan et al, 2020) control facial expression and posture in the original StyleGAN image domain, and others (Richardson et al, 2020;Wang et al, 2021) deal well with the difficult task of facial super-resolution.…”