Designing an Encoder for StyleGAN Image Manipulation

Tov, Omer; Alaluf, Yuval; Nitzan, Yotam; Patashnik, Or; Cohen–Or, Daniel

doi:10.48550/arxiv.2102.02766

Cited by 33 publications

(77 citation statements)

References 34 publications

(98 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…• GAN Inversion employs the state-of-the-art GAN inversion method (Tov et al, 2021) to obtain the latent codes for real images. In more detail, we map the annotated real images to the GAN latent space, which allows computing the intermediate generator activations and using them as pixel-level representations.…”

Section: Methodsmentioning

confidence: 99%

Label-Efficient Semantic Segmentation with Diffusion Models

Baranchuk¹,

Ivan²,

Voynov³

et al. 2021

Preprint

View full text Add to dashboard Cite

Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision. The source code of the project is publicly available.

show abstract

Section: Methodsmentioning

confidence: 99%

Label-Efficient Semantic Segmentation with Diffusion Models

Baranchuk¹,

Ivan²,

Voynov³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In order to benefit from these properties in real images, it is necessary to obtain the latent code from which a pretrained GAN can reconstruct the original input image. This task, commonly referred to as GAN Inversion, has been tackled by numerous recent works, either by using: (i) optimization (Abdal et al, 2019;Karras et al, 2020b); or (ii) an encoder (Guan et al, 2020;Pidhorskyi et al, 2020;Richardson et al, 2021;Tov et al, 2021); or (iii) a hybrid approach using both (Zhu et al, 2016;Baylies, 2019;Zhu et al, 2020a). See Xia et al (2021) for a more thorough review.…”

Section: Related Workmentioning

confidence: 99%

StyleAlign: Analysis and Applications of Aligned StyleGAN Models

Wu¹,

Nitzan²,

Shechtman³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we perform an in-depth study of the properties and applications of aligned generative models. We refer to two models as aligned if they share the same architecture, and one of them (the child) is obtained from the other (the parent) via fine-tuning to another domain, a common practice in transfer learning. Several works already utilize some basic properties of aligned StyleGAN models to perform image-to-image translation. Here, we perform the first detailed exploration of model alignment, also focusing on StyleGAN. First, we empirically analyze aligned models and provide answers to important questions regarding their nature. In particular, we find that the child model's latent spaces are semantically aligned with those of the parent, inheriting incredibly rich semantics, even for distant data domains such as human faces and churches. Second, equipped with this better understanding, we leverage aligned models to solve a diverse set of tasks. In addition to image translation, we demonstrate fully automatic cross-domain image morphing. We further show that zero-shot vision tasks may be performed in the child domain, while relying exclusively on supervision in the parent domain. We demonstrate qualitatively and quantitatively that our approach yields state-of-the-art results, while requiring only simple fine-tuning and inversion.

show abstract

“…Manipulating the latent representation in StyleGAN can be used to edit the semantics of the synthesized images. Real images also can be inverted into the latent spaces of StyleGAN by GAN Inversion methods [1,2,23,29,37], enabling the further manipulations. The approaches of semantic image editing with StyleGAN can be roughly divided into two groups, i.e., supervised approaches and unsupervised approaches.…”

Section: Related Workmentioning

confidence: 99%

“…We first invert the input images to the latent codes with the GAN inversion method for further editing. More concretely, we use the e4e encoder [29] to embed the given input image into the W+ space, and then manipulate multiple target attributes in turn. Fig .…”

Section: Manipulation Of Real Imagesmentioning

confidence: 99%

See 1 more Smart Citation

Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation

Wang

Chen

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Image manipulation with StyleGAN has been an increasing concern in recent years. Recent works have achieved tremendous success in analyzing several semantic latent spaces to edit the attributes of the generated images. However, due to the limited semantic and spatial manipulation precision in these latent spaces, the existing endeavors are defeated in fine-grained StyleGAN image manipulation, i.e., local attribute translation. To address this issue, we discover attribute-specific control units, which consist of multiple channels of feature maps and modulation styles. Specifically, we collaboratively manipulate the modulation style channels and feature maps in control units rather than individual ones to obtain the semantic and spatial disentangled controls. Furthermore, we propose a simple yet effective method to detect the attribute-specific control units. We move the modulation style along a specific sparse direction vector and replace the filter-wise styles used to compute the feature maps to manipulate these control units. We evaluate our proposed method in various face attribute manipulation tasks. Extensive qualitative and quantitative results demonstrate that our proposed method performs favorably against the state-of-the-art methods. The manipulation results of real images further show the effectiveness of our method. CCS CONCEPTS• Computing methodologies → Image manipulation.

show abstract

Designing an Encoder for StyleGAN Image Manipulation

Abstract: Figure 1: Real image editing via StyleGAN inversion using our e4e method. For each domain we show from left to right: the original real image, the inverted image, and multiple manipulations performed using various editing techniques.

Cited by 33 publications

References 34 publications

Label-Efficient Semantic Segmentation with Diffusion Models

Label-Efficient Semantic Segmentation with Diffusion Models

StyleAlign: Analysis and Applications of Aligned StyleGAN Models

Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation

Contact Info

Product

Resources

About