GeneGAN: Learning Object Transfiguration and Object Subspace from Unpaired Data

Abstract

Object Transfiguration generates diverse novel images by replacing an object in the given image with particular objects from exemplar images. It offers fine-grained controls of image generation, and can perform tasks like "put exactly those eyeglasses from image A onto the nose of the person in image B". However, object transfiguration often requires disentanglement of objects from backgrounds in feature space, which is challenging and previously requires learning from paired training data: two images sharing the same background but with different objects. In this work, we propose a deterministic generative model that learns disentangled feature subspaces by adversarial training. The training data are two unpaired sets of images: a positive set containing images that have some kind of object, and a negative set being the opposite. The model encodes an image into two complement features: one for the object, and the other for the background. The object and background features from a "positive" parent and a "negative" parent, can be recombined to produce four children, of which two are exact reproductions, and the other two are crossbreeds. Minimizing the adversarial loss between crossbreeds and parents will ensure the crossbreeds inherit the specific objects of parents. On the other hand, minimizing the reconstruction loss between reproductions and parents can ensure the completeness of the features. Overall, the object and background features are complete and disentangled representations of images. Moreover, the object features are found to constitute a multidimensional attribute subspace. Experiments on CelebA and Multi-PIE datasets validate the effectiveness of the proposed model on real world data, for generating images with specified eyeglasses, smiling, hair styles, and lighting conditions.

Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers

“…These have been incorporated into the several attribute swapping generative models. Zhou et al [73] recombine the information of the latent information of two images to swap a specific attribute between the given images. Liu et al [36] generate high quality images by coupling GANs in order to learn a shared latent representation in order to tackle several unsupervised image translation tasks including domain adaptation and face image translation.…”

A Survey of Deep Facial Attribute Analysis

“…Zhou et al [136] first design a GeneGAN to achieve the basic reference exemplar-based facial attribute manipulation. Given an image, it is encoded into two complement codes: attribute-specific codes and attributeirrelevant codes.…”

How Generative Adversarial Networks and Their Variants Work

“…Object transfiguration is a conditional image generation that replaces an object in an image with a particular condition while the background does not change. Zhou et al [144] adopted an encoder-decoder structure to transplant an object, where the encoder decomposes an image into the background feature and the object feature, and the decoder reconstructs the image from the background feature and the object feature we want to transfigure. Importantly, to disentangle the encoded feature space, two separated training sets are required where one is the set of images having the object and the other is the set of images not having the object.…”

“…Their capability to represent complex and high-dimensional data can be utilized in treating images [2,12,57,65,127,133,145], videos [122,125,126], music generation [41,66,141], natural languages [48,73] and other academic domains such as medical images [16,77,136] and security [109,124]. Specifically, generative models are highly useful for image to image translation (See Figure 1) [9,57,137,145] which transfers images to another specific domain, image super-resolution [65], changing some features of an object in an image [3,37,75,94,144] and predicting the next frames of a video [122,125,126]. In addition, generative models can be the solution for various problems in the machine learning field such as semisupervised learning [21,67,104,115], which tries to address the lack of labeled data, and domain adaptation [2,12,47,108,111,140], which leverages known knowled...…”