Realistic Image Generation using Region-phrase Attention

Abstract

The Generative Adversarial Network (GAN) has recently been applied to generate synthetic images from text. Despite significant advances, most current state-of-the-art algorithms are regular-grid region based; when attention is used, it is mainly applied between individual regular-grid regions and a word. These approaches are sufficient to generate images that contain a single object in its foreground, such as a "bird" or "flower". However, natural languages often involve complex foreground objects and the background may also constitute a variable portion of the generated image. Therefore, the regular-grid based image attention weights may not necessarily concentrate on the intended foreground region(s), which in turn, results in an unnatural looking image. Additionally, individual words such as "a", "blue" and "shirt" do not necessarily provide a full visual context unless they are applied together. For this reason, in our paper, we proposed a novel method in which we introduced an additional set of attentions between truegrid regions and word phrases. The true-grid region is derived using a set of auxiliary bounding boxes. These auxiliary bounding boxes serve as superior location indicators to where the alignment and attention should be drawn with the word phrases. Word phrases are derived from analysing Part-of-Speech (POS) results. We perform experiments on this novel network architecture using the Microsoft Common Objects in Context (MSCOCO) dataset and the model generates 256 × 256 conditioned on a short sentence description. Our proposed approach is capable of generating more realistic images compared with the current state-ofthe-art algorithms.

Region-Guided CycleGANs for Stain Transfer in Whole Slide Images

“…In [17], pedestrians are edited into predefined scenes using pix2pix, with spatial pyramid pooling in the discriminator for direct scrutiny. Pre-trained R-CNN object detection systems have been used to propose regions during GAN training in [8], or as a feature extractor in object-driven GANs [12]. In contrast, we modify the GAN discriminator itself, and leverage automatically derived RoI data.…”

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

“…The work of Huang et al [129] improved the DAMSM loss by introducing true-grid regions inside every bounding box with word phrases, where attention weights depend on the bounding box and phrase information. So, this mechanism extends the regular gridbased attention that utilizes additional phrase features through parts-of-speech tagging besides sentence and word features.…”

“…Similar to [129], Dynamic Aspect-awarE GAN (DAE-GAN) [136] refers to the importance of aspect in the input text. The model represents text information from multiple granularities of sentence-level, word-level, and aspect-level, for which, besides other attention mechanisms, the aspect-aware dynamic re-drawer (ADR) module is employed.…”