Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network

Abstract

This paper presents a novel method to deal with the challenging task of generating photographic images conditioned on semantic image descriptions. Our method introduces accompanying hierarchical-nested adversarial objectives inside the network hierarchies, which regularize mid-level representations and assist generator training to capture the complex image statistics. We present an extensile single-stream generator architecture to better adapt the jointed discriminators and push generated images up to high resolutions. We adopt a multi-purpose adversarial loss to encourage more effective image and text information usage in order to improve the semantic consistency and image fidelity simultaneously. Furthermore, we introduce a new visual-semantic similarity measure to evaluate the semantic consistency of generated images. With extensive experimental validation on three public datasets, our method significantly improves previous state of the arts on all datasets over different evaluation metrics.

Facial Synthesis From Visual Attributes via Sketch Using Multiscale Generators

“…For fair comparison, the multi-scale resolution settings are the same as used with the experiments on the CelebA dataset. In particular, we use 32 × 32 and 64 × 64 resolution scales for StackGAN [15] training and 16 × 16, 32 × 32 and 64 × 64 multiple resolution scales for HDGAN [17] and StackGAN++ [16] as well as our proposed method. The disCVAE method produces reconstructions which are blurry.…”

“…Previous conditional GAN-based approaches such as GAN-INT-CLS [14] and StackGAN [15] also produce poor quality results due to the model collapse during training. Recent StackGAN++ and HDGAN works generate plausible facial images (HDGAN is better at the color Methods FID score HDGAN [17] 114.912 StackGAN++ [16] 35.988 Single-scale (proposed method) 37.381 Proposed method 30.566 diversity). The previous work Attribute2Sketch2Face, which is a combination of CVAE and GAN, is also able to generate facial images with corresponding attributes.…”

“…The CelebA-HQ dataset [50] is a high-quality version of the CelebA dataset, which consists of 30,000 images with 1024 × 1024 resolution. Due to GPU and memory limitations, we conduct experiments on 256 × 256 resolution images and compare the performance with StackGAN++ [16] and HDGAN [17]. The reason why we chose these two baselines is due to their capability to deal with high resolution images.…”

“…Xu et al [18] proposed an attentiondriven method to improve the synthesis results. Zhang et al [17] (HDGAN) adopted a multi-adversarial loss to improve the synthesis by leveraging more effective image and text information at multi-scale layers.…”

“…4. In order to learn the discrimination in both image content and semantics, we adopt the triplet matching training strategy [14], [16], [17], [53]. Specifically, given sketch attributes, the discriminator is trained by using the following triplets: (i) real-sketch and real-sketch-attributes, (ii) synthesized-sketch and real-sketch-attributes, and (iii) wrong-sketch (real sketch but mismatching attributes) and same real-sketch-attributes.…”