Modular Generative Adversarial Networks

Abstract

Input Black no smile Blond Brown No Smile smile male female + no smile + male + male + no smile + male + smile + female + female + smile + female Brown Hair Gender Expression Smile Brown Hair Brown Hair Hair Color ModularGAN ArchitectureFig. 1. ModularGAN: Results of proposed modular generative adversarial network illustrated on multi-domain image-to-image translation task on the CelebA [1] dataset.Abstract. Existing methods for multi-domain image-to-image translation (or generation) attempt to directly map an input image (or a random vector) to an image in one of the output domains. However, most existing methods have limited scalability and robustness, since they require building independent models for each pair of domains in question. This leads to two significant shortcomings: (1) the need to train exponential number of pairwise models, and (2) the inability to leverage data from other domains when training a particular pairwise mapping. Inspired by recent work on module networks [2], this paper proposes ModularGAN for multi-domain image generation and image-to-image translation. Mod-ularGAN consists of several reusable and composable modules that carry on different functions (e.g., encoding, decoding, transformations). These modules can be trained simultaneously, leveraging data from all domains, and then combined to construct specific GAN networks at test time, according to the specific image translation task. This leads to Modu-larGAN's superior flexibility of generating (or translating to) an image in any desired domain. Experimental results demonstrate that our model not only presents compelling perceptual results but also outperforms state-of-the-art methods on multi-domain facial attribute transfer.

How Good Is My GAN?

“…Since the original GAN model [19] was proposed, many variants have appeared in the past few years, for example, to improve the quality of the generated images [12,15,25,36], or to stabilize the training procedure [7,9,20,34,36,40,57]. GANs have also been modified to generate images of a given class by conditioning on additional information, such as the class label [16,35,37,41].…”

How to Make a Pizza: Learning a Compositional Layer-Based GAN Model

“…Our work is also related to studies investigating the modularity and the composability of GANs [17,53]. Recently, Zhao et al [53] proposed a modular multidomain GAN architecture, which consists of several composable modular operations. However, they assume that all the operations are order-invariant which cannot be true for adding and removing overlapping objects in an image.…”

“…Generating residual images. Recently, researchers have explored the idea of using a cGAN model to generate only residual images, i.e., only the part of the image that needs to be changed when it is translated into another domain, for the task of face manipulation [35,40,53]. For example, these models are able to learn how to change the hair color, open/close the mouth, or change facial expressions by manipulating only the corresponding parts of the faces.…”

Fast Face Image Synthesis With Minimal Training

“…Face image synthesis has been a popular research area recently [41], mainly as a means to generate artificial training samples for CNNs [38]. Generative adversarial nets (GANs) [21] have made tremendous progress in this domain with different GAN models being used to generate synthetic face images with different pose [58,9,68], facial feature [52,10,60,67], age [20,3] and expression [39]. However, as pointed out by Karras et al [33], GANs require plenty of training data (10M images), time (19 days) and GPU resources to generate high quality (1024×1024) face images.…”