GAN Inversion: A Survey

Abstract

GAN inversion aims to invert a given image back into the latent space of a pretrained GAN model, for the image to be faithfully reconstructed from the inverted code by the generator. As an emerging technique to bridge the real and fake image domains, GAN inversion plays an essential role in enabling the pretrained GAN models such as StyleGAN and BigGAN to be used for real image editing applications. Meanwhile, GAN inversion also provides insights on the interpretation of GAN's latent space and how the realistic images can be generated. In this paper, we provide an overview of GAN inversion with a focus on its recent algorithms and applications. We cover important techniques of GAN inversion and their applications to image restoration and image manipulation. We further elaborate on some trends and challenges for future directions. A curated list of GAN inversion methods, datasets, and other related information can be found at github.com/weihaox/awesome-gan-inversion.

LTT-GAN: Looking Through Turbulence by Inverting GANs

“…GAN Inversion for Restoration with well-trained GANs can provide photo-realistic priors, and thus it is a new promising avenue for image restoration. According to the recent survey of GAN inversion [45], there are two major ways for embedding a well-trained GAN into the restoration framework. Typical methods [10,14,26,30] belong to the first approach in which the optimal latent code of GANs are retrieved with some iterative optimization algorithms.…”

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

“…Besides producing impressive image samples, generative adversarial networks (GANs) [9] have been shown to learn meaningful latent spaces [18] with extensive studies on multiple derived spaces [15,44] and various knobs and controls for conditional human face generation [12,28,42]. Encoding an image to the GAN's latent space requires an optimization-based inversion process [19,45] or an external image encoder [30], which has limited reconstruction fidelity (or produces latent codes in much higher dimensions outside the learned manifold). This problem may be related to the GAN's limited latent size and mode-collapse problem, where the latent space only partially covers the support of training samples.…”

“…However, the resulting latent variable lacks high-level semantics and other desirable properties, such as disentanglement, compactness, or the ability to perform meaningful linear interpolation in the latent space. Alternatively, one can use a trained GAN for extracting a representation using the so-called GAN inversion [19,45], which optimizes for a latent code that reproduces the given input. Even though the resulting code carries rich semantics, this technique struggles to faithfully reconstruct the input image.…”

Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

“…State-of-the-art methods, such as PGGAN [18], BigGAN [6], StyleGAN [20], and StyleGAN2 [21], employ large-scale training with contemporary techniques, achieving photorealistic results. These methods have been extended to various tasks, including face generation [18,20,21], image editing [1,8,37], semantic image synthesis [48,36,29], image-to-image translation [14,58,9,17,16,40], style transfer [28,13,26], and GAN inversion [35,40,51]. Despite the remarkable success, the performance of GANs relies heavily on the amount of training data.…”