Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation

Abstract

Image manipulation with StyleGAN has been an increasing concern in recent years. Recent works have achieved tremendous success in analyzing several semantic latent spaces to edit the attributes of the generated images. However, due to the limited semantic and spatial manipulation precision in these latent spaces, the existing endeavors are defeated in fine-grained StyleGAN image manipulation, i.e., local attribute translation. To address this issue, we discover attribute-specific control units, which consist of multiple channels of feature maps and modulation styles. Specifically, we collaboratively manipulate the modulation style channels and feature maps in control units rather than individual ones to obtain the semantic and spatial disentangled controls. Furthermore, we propose a simple yet effective method to detect the attribute-specific control units. We move the modulation style along a specific sparse direction vector and replace the filter-wise styles used to compute the feature maps to manipulate these control units. We evaluate our proposed method in various face attribute manipulation tasks. Extensive qualitative and quantitative results demonstrate that our proposed method performs favorably against the state-of-the-art methods. The manipulation results of real images further show the effectiveness of our method. CCS CONCEPTS• Computing methodologies → Image manipulation.

PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs

“…However, whilst several methods identify ways of manipulating the latent space of GANs to bring about global semantic changes-either in a supervised [16,44,48,47] or unsupervised [53,49,20,52,42] manner-many of them struggle to apply local changes to regions of interest in the image. In this framework of local image editing, one can swap certain parts between images [11,24,8,50,32,2], or modify the style at particular regions [54,55,5,63,62,39,26]. This is achieved with techniques such as clustering [11,62,5,26], manipulating the AdaIN [23] parameters [55,54], or/and operating on the feature maps themselves [54,5,62] to aid the locality of the edit.…”

“…In this framework of local image editing, one can swap certain parts between images [11,24,8,50,32,2], or modify the style at particular regions [54,55,5,63,62,39,26]. This is achieved with techniques such as clustering [11,62,5,26], manipulating the AdaIN [23] parameters [55,54], or/and operating on the feature maps themselves [54,5,62] to aid the locality of the edit. Other approaches employ additional latent spaces or architectures [32,39], require the computation of expensive gradient maps [54,55] and semantic segmentation masks/networks [55,64,39], or require manually specified regions of interest [63].…”

“…This is achieved with techniques such as clustering [11,62,5,26], manipulating the AdaIN [23] parameters [55,54], or/and operating on the feature maps themselves [54,5,62] to aid the locality of the edit. Other approaches employ additional latent spaces or architectures [32,39], require the computation of expensive gradient maps [54,55] and semantic segmentation masks/networks [55,64,39], or require manually specified regions of interest [63]. In contrast to related work, our method automatically learns both the parts and a diverse set of global appearances, in a fast unsupervised procedure without any semantic masks.…”

“…One important question in this line of research is how latent representations are combined to form the appearance at a particular local region of the image. Whilst some recent methods attempt to tackle this problem [54,55,5,63,62,39,26], the current state-of-the-art methods come with a number of important drawbacks and limitations. In particular, existing techniques require prohibitively long training times [55,63], costly Jacobian-based optimization [63], and the requirement of semantic masks [55] or manually specified regions of interest [63].…”

GAN-Based Face Attribute Editing

“…In StyleSpace-Analysis [92], the translation vector is computed in the S space for better disentanglement, and much fewer samples are required (e.g., to compute n S . ACU [169] extends StyleSpaceAnalysis by also manipulating feature maps in the generator, which can achieve more realistic FAM results without damaging the spatial disentanglement of image changes. On the other hand, InterFaceGAN [33] models n Z as the normal vector of hyper-planes separating latent codes corresponding to different attribute labels.…”