EditGAN: High-Precision Semantic Image Editing

Abstract

Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN-based image editing methods often require large-scale datasets with semantic segmentation annotations for training, only provide high level control, or merely interpolate between different images. Here, we propose EditGAN, a novel method for high-quality, high-precision semantic image editing, allowing users to edit images by modifying their highly detailed part segmentation masks, e.g., drawing a new mask for the headlight of a car. EditGAN builds on a GAN framework that jointly models images and their semantic segmentations [1, 2], requiring only a handful of labeled examples -making it a scalable tool for editing. Specifically, we embed an image into the GAN's latent space and perform conditional latent code optimization according to the segmentation edit, which effectively also modifies the image. To amortize optimization, we find "editing vectors" in latent space that realize the edits. The framework allows us to learn an arbitrary number of editing vectors, which can then be directly applied on other images at interactive rates. We experimentally show that EditGAN can manipulate images with an unprecedented level of detail and freedom, while preserving full image quality.We can also easily combine multiple edits and perform plausible edits beyond EditGAN's training data. We demonstrate EditGAN on a wide variety of image types and quantitatively outperform several previous editing methods on standard editing benchmark tasks. Project page: https://nv-tlabs.github.io/editGAN.

FlexIT: Towards Flexible Semantic Image Translation

“…Deep generative networks, like GANs, have given rise to numerous image editing applications, ranging from photography retouching [43], image inpainting [54], object insertion [17], domain translation [55,58], colorization [24], super-resolution [26,36], among many others. Automatic user-driven image editing aims at providing the user control to modify an image, by tweaking segmentation masks [38], scene graphs [10], or class labels [6]. Allowing the user to provide unstructured free-form text queries is more challenging.…”

Representation Learning for Semantic Scene Understanding

“…SESAME [47] enables users to draw a mask with semantic labels on an image to indicate the category of changed pixels. Similarly, EditGAN [48] allows users to alter object appearance by modifying a detailed object part segmentation map [49,50]. SIMSG [23] employs scene graphs as the interface, where users can manipulate images by altering the nodes or edges of a graph.…”

Les GANs : stimulateurs de créativité en phase d’idéation

“…Une application importante de cette technique est le semantic GAN, avec notamment : SPADE [25], opérant une synthèse d'image conditionnée par une image label avec des classes sémantiques, ou GauGAN : une application de sketching temps réel qui génère des images de paysages réalistes à partir de dessins d'utilisateurs. Les papiers les plus récents sur les semantic GANs se trouvent dans [20,28].…”

“…La réédition d'image par inversion du GAN est une technique puissante pour transférer dans l'espace latent l'approximation d'une image et explorer la capacité générative autour d'un concept importé. Ces approches vont conduire à une formalisation sémantique semi-supervisée qui devrait intéresser les architectes puisqu'elle va leur permettre de sélectionner, nommer et trier des propriétés implicitement apprises dans l'espace latent, puis d'éditer des objets réels en manipulant ces propriétés comme de simples variables en modélisation paramétrique [1,20,31,37,38].…”