Object-Driven Text-To-Image Synthesis via Adversarial Training

Abstract

In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes. Following the two-step (layout-image) generation process, a novel object-driven attentive image generator is proposed to synthesize salient objects by paying attention to the most relevant words in the text description and the pre-generated semantic layout. In addition, a new Fast R-CNN based object-wise discriminator is proposed to provide rich object-wise discrimination signals on whether the synthesized object matches the text description and the pre-generated layout. The proposed Obj-GAN significantly outperforms the previous state of the art in various metrics on the large-scale COCO benchmark, increasing the Inception score by 27% and decreasing the FID score by 11%. A thorough comparison between the traditional grid attention and the new object-driven attention is provided through analyzing their mechanisms and visualizing their attention layers, showing insights of how the proposed model generates complex scenes in high quality.

Image Synthesis From Reconfigurable Layout and Style

“…For pixels where bounding boxes of different objects overlap, their semantic labels are assigned by objects with the highest predicted mask weight. Unlike [12,21] where ground truth masks is adopted to guide learning of shape generator, our model can learn semantic masks in a weakly-supervised manner. Even for objects with overlapped bounding box, like person and surfboard in (f), synthesized images and learned masks are consistent and semantically reasonable.…”

“…Spatial layout conditioned image generation has been studied in recent literature. In [16,12,11,21], layout and object information is utilized in text-to-image generation. [11] controls location of multiple objects in text-to-image generation by adding an object pathway to both the generator and discriminator.…”

“…[16,12,21] performs text-to-image synthesis in two steps: semantic layout (class label and bounding boxes) generation from text first, and image synthesis conditioned on predicted semantic layout and text description. However, [12,21] requires pixel-level instance segmentation annotation, which is labor intensive to collect, for training of shape generator, while our method does not require pixel-level annotation and can learn segmentation mask in a weaklysupervised manner. [38] studied similar task with us, where variational autoencoders based network is adopted for scene image generation from layout.…”

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

“…Complementary image features derived from different models, such as ResNet and Faster R-CNN, are used for multiple image attention mechanisms [139]. Moreover, the reverse of image attention that generates attended text feature with image and text input is used for text-to-image synthesis in [48] and [140].…”

LayoutVAE: Stochastic Scene Layout Generation From a Label Set

“…Despite the rapid progress and recent successes in object generation (e.g., celebrity face, animals, etc.) [1,9,13] and scene generation [4,11,12,19,22,30,31], little attention has been paid to frameworks designed for stochastic semantic layout generation. Having a robust model for layout generation will not only allow us to generate reliable scene layouts, but also provide priors and means to infer latent relationships between objects, advancing progress in the scene understanding domain.…”