TediGAN: Text-Guided Diverse Face Image Generation and Manipulation

Abstract

In this work, we propose TediGAN, a novel framework for multi-modal image generation and manipulation with textual descriptions. The proposed method consists of three components: StyleGAN inversion module, visual-linguistic similarity learning, and instance-level optimization. The inversion module is to train an image encoder to map real images to the latent space of a well-trained StyleGAN. The visual-linguistic similarity is to learn the text-image matching by mapping the image and text into a common embedding space. The instance-level optimization is for identity preservation in manipulation. Our model can provide the lowest effect guarantee, and produce diverse and highquality images with an unprecedented resolution at 1024 2 . Using a control mechanism based on style-mixing, our Tedi-GAN inherently supports image synthesis with multi-modal inputs, such as sketches or semantic labels with or without instance (text or real image) guidance. To facilitate text-guided multi-modal synthesis, we propose the MULTI-MODAL CELEBA-HQ, a large-scale dataset consisting of real face images and corresponding semantic segmentation map, sketch, and textual descriptions. Extensive experiments on the introduced dataset demonstrate the superior performance of our proposed method. Code and data are available at https://github.com/weihaox/TediGAN.

Cycle-Consistent Inverse GAN for Text-to-Image Synthesis

“…One reason is that we cannot get the inversion results that are the same as the original images, then in the w ′ optimization process the model loses part of the original semantic information. Another reason is that the latent space W of StyleGAN is still entangled with some semantic attributes [33], which means that when we change part of the attributes, the rest attributes will also be affected. We leave improving the space W with better disentanglement for the future work.…”

“…This method typically reconstruct the original images by optimizing the latent codes, which is based on either gradient descent [40] or other iterative algorithms [12]. Some works [4,33,40] attempt to combine these two ideas, where the produced latent codes by the GAN inversion encoder are used as the initialization for the optimization step, as it is hard to get the perfect reconstructed images based on the inverted latent codes from a single GAN inversion encoder. Zhu et al [40] propose to train the encoder based on the real images instead of the fake images, as a result, the trained encoder can be better adapted to the real scenarios.…”

“…In our work, we combine the learning-based and optimizationbased approaches to find the latent codes with the desired attributes. Existing works of GAN inversion mainly focus on the face image datasets [33,40], while we target two more challenging datasets in the wild, where the Recipe1M dataset contains cooked food images with multiple mixed ingredients and the CUB dataset has birds with various poses, shapes, colours and backgrounds. The difference between our proposed method and the existing works is that we use the cycle consistency training method, applying regularization on not only the image domain, but also the latent space domain.…”

“…For the GAN inversion task, training an encoder 𝐸 (•) is a practical solution [33,40], which takes the images as the input, and outputs w ′ aligned with the original w. To allow the reconstructed image from w ′ to have the same semantic information as the original image, we train the encoder 𝐸 (•) with the pixel losses and the perceptual loss [11]. Note that we use the real images x as inputs for 𝐸 (•) training, in an attempt to make the encoder more applicable to real image manipulation.…”

Embedding Arithmetic for Text-driven Image Transformation

“…Zheng et al (2020a) encode images as a graph of interacting objects which lets the user modify an image by editing its scene graph. GANs are also frequently used to modify images based on some natural language input (Nam et al, 2018;Li et al, 2020a;Xia et al, 2020). Lastly, CLIP (Radford et al, 2021) can be used in combination with a StyleGAN generator to make semantic edits in images, as exemplified in (Patashnik et al, 2021).…”