Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

Abstract

We propose a novel hierarchical approach for text-toimage synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple steps, in which it first constructs a semantic layout from the text by the layout generator and converts the layout to an image by the image generator. The proposed layout generator progressively constructs a semantic layout in a coarse-to-fine manner by generating object bounding boxes and refining each box by estimating object shapes inside the box. The image generator synthesizes an image conditioned on the inferred semantic layout, which provides a useful semantic structure of an image matching with the text description. Our model not only generates semantically more meaningful images, but also allows automatic annotation of generated images and user-controlled generation process by modifying the generated scene layout. We demonstrate the capability of the proposed model on challenging MS-COCO dataset and show that the model can substantially improve the image quality, interpretability of output and semantic alignment to input text over existing approaches.

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

“…[212] includes extra object pathway to both generator and discriminator to explicit control the object locations. [213] employs a two-stage procedure that first builds a semantic layout automatically from the input sentence with LSTM based box and shape generators, and then synthesizes the image using image generator and discriminators. Since fine-grained word/object level information is not explicitly used for generation, such synthesized images do not contain enough details to make them look realistic.…”

LayoutVAE: Stochastic Scene Layout Generation From a Label Set

“…Reed et al [24] perform image generation with sentence input along with additional information in the form of keypoints or bounding boxes. Hong et al [11] break down the process of generating an image from a sentence into multiple stages. The input sentence is first used to predict the objects that are present in the scene, followed by prediction of bounding boxes, then semantic segmentation masks, and finally the image.…”

“…Despite the rapid progress and recent successes in object generation (e.g., celebrity face, animals, etc.) [1,9,13] and scene generation [4,11,12,19,22,30,31], little attention has been paid to frameworks designed for stochastic semantic layout generation. Having a robust model for layout generation will not only allow us to generate reliable scene layouts, but also provide priors and means to infer latent relationships between objects, advancing progress in the scene understanding domain.…”

Seq-SG2SL: Inferring Semantic Layout From Scene Graph Through Sequence to Sequence Learning

“…As clip arts in abstract scene can be easily generalized to object bounding boxes in semantic layout, this concept extends to real images [20]. Predicting a semantic layout from text is usually posed as an intermediate step for complex image generation [7] [9]. A complex image refers to the one containing multiple interactive objects.…”

“…Figure 1: The Seq-SG2SL framework for inferring semantic layout from scene graph. scene graph [11] [13] for semantic description and semantic layout [9] [26] for image. Therefore, our goal in this work solves the underlying task, inferring semantic layout from scene graph, for connecting text to image.…”