Semantic Object Accuracy for Generative Text-to-Image Synthesis

Abstract

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are mentioned in the image caption, e.g. whether an image generated from "a car driving down the street" contains a car. We perform a user study comparing several text-to-image models and show that our SOA metric ranks the models the same way as humans, whereas other metrics such as the Inception Score do not. Our evaluation also shows that models which explicitly model objects outperform models which only model global image characteristics.

GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks

“…In the field of text-to-image synthesis, Hinz et al [61] introduce Semantic Object Accuracy (SOA) to evaluate images given an image caption. LayoutGAN is proposed by Li et al [62] for graphic design and scene generation, introducing wireframe rendering for image discrimination.…”

FA-GAN: Feature-Aware GAN for Text to Image Synthesis

“…In other words, if the model generates the same image, the FID will be higher (the lower the FID, the better), but IS can not penalize this case. [11,12,13] found that IS is not an appropriate metric to evaluate the text-to-image synthesis models since some models tend to generate the same image when the text contains the same word, which is not good generative models but IS could be high (the higher the IS, the better). Thus, we use FID to evaluate our models.…”

“…Most existing works [8,9,10,11,12] have achieved remarkable progress by proposing effective structures of GANs. StackGAN [8] uses the stacked structure of multiple GANs to decompose the hard problem of generating highresolution images into tractable subproblems.…”

“…StackGAN [8] uses the stacked structure of multiple GANs to decompose the hard problem of generating highresolution images into tractable subproblems. Subsequent studies [9,10,11,12] have refined the architecture based on StackGAN [8]. AttnGAN [9] adopts cross-modal attention mechanisms for fine-grained generation.…”

“…DM-GAN [10] leverages dynamic memory modules to supplement the generation procedure. Obj-GAN [11] and OP-GAN [12] use an additional input, pre-generated scene layout, to concentrate on creating objects. However, it is not easy to train multiple GANs at one time.…”