On the "steerability" of generative adversarial networks

Abstract

Figure 1: Learned latent space trajectories in generative adversarial networks correspond to visual transformations like camera shift and zoom. These transformations can change the distribution of generated data, but only so much -biases in the data, like centered objects, reveal themselves as objects get "stuck" at the image borders when we try to shift them out of frame. Take the "steering wheel", drive in the latent space, and explore the natural image manifold via generative transformations!

Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation

“…The approaches of semantic image editing with StyleGAN can be roughly divided into two groups, i.e., supervised approaches and unsupervised approaches. The supervised approaches [3,9,13,14,19,25] introduce pretrained classifiers to find the directions that alter the output of the classifiers. For example, InterfaceGAN [1] trains linear support vector machines (SVMs) using the attribute annotations labeled by the off-the-shelf classifiers and finds hyperplanes in the latent space serving as the separation boundary.…”

“…GAN Dissection [5] found the causal feature maps are specialized to synthesize specific visual arXiv:2111.13010v1 [cs.CV] 25 Nov 2021 concepts in generated images. Some previous works [9,10,14,32] find that rich semantic information is encoded in the latent space of StyleGAN, e.g., Z or W space and various semantic manipulations can be achieved by moving the latent code along the direction in the latent space. But modifications to the latent code in the Z or W space are spatial entangled.…”

Optimizing Latent Space Directions For GAN-based Local Image Editing

“…Zhu et al [17] Unsupervised [19,14] Self-Supervised InterFaceGAN [9] Supervised GANALYZE [13] Supervised StyleSpace [16] Supervised LELSD (Ours) Supervised Image Composition Bau et al [20] Unsupervised Chai et al [21] Unsupervised Editing in Style [15] Unsupervised Zhang et al [7] Supervised Barbershop [22] Supervised not only agnostic to the GAN architecture, but also is able to effectively disentangle the semantic attributes. To this end, we propose Locally Effective Latent Space Directions (LELSD), a framework to find the latent space directions that affect local regions of the output image.…”

“…To solve this problem, [9,13,14] use an external supervision and find latent space directions that yield the desired change in the generated images. This is done by finding the latent space direction that maximizes a designed objective function.…”

“…Several recent works control the semantics of the GANgenerated image by tweaking the latent code to perform global [9,10,11,12,13,14] or localized [7,15,16,17] editing. Although a lot of progress has been made in global image editing, it remains challenging to disentangle the semantic attributes and thus control the local semantics of the image.…”