DRAW: A Recurrent Neural Network For Image Generation

Gregor, Karol; Danihelka, Ivo; Graves, Alex; Rezende, Danilo Jimenez; Wierstra, Daan

doi:10.48550/arxiv.1502.04623

Cited by 242 publications

(295 citation statements)

References 0 publications

Supporting

Mentioning

295

Contrasting

Order By: Relevance

“…Dataset: We evaluate our method using cluttered MNIST [6] and cluttered Fashion-MNIST datasets. cluttered MNIST has been used by several works to demonstrate visual attention [31,6]. We followed the same procedure for generating cluttered Fashion-MNIST which includes ten clothing categories.…”

Section: Methodsmentioning

confidence: 99%

A Reinforcement Learning Approach for Sequential Spatial Transformer Networks

Azimi

Raue

Hees

et al. 2019

Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation

View full text Add to dashboard Cite

Spatial Transformer Networks (STN) can generate geometric transformations which modify input images to improve classifier's performance. In this work, we combine the idea of STN with Reinforcement Learning (RL). To this end, we break the affine transformation down into a sequence of simple and discrete transformations. We formulate the task as a Markovian Decision Process (MDP) and use RL to solve this sequential decision making problem. STN architectures learn the transformation parameters by minimizing the classification error and backpropagating the gradients through a sub-differentiable sampling module. In our method, we are not bound to differentiability of the sampling modules. Moreover, we have freedom in designing the objective rather than only minimizing the error; e.g., we can directly set the target as maximizing the accuracy. We design multiple experiments to verify the effectiveness of our method using cluttered MNIST and Fashion-MNIST datasets and show that our method outperforms STN with proper definition of MDP components.

show abstract

Section: Methodsmentioning

confidence: 99%

A Reinforcement Learning Approach for Sequential Spatial Transformer Networks

Azimi

Raue

Hees

et al. 2019

Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation

View full text Add to dashboard Cite

show abstract

“…Due to the great potential in broad applications, textto-image synthesis though challenging yet is arousing extensive research attention. Earlier methods have achieved progress on this task due to the emergence of deep generative models [14,6,15,30,21].…”

Section: Related Workmentioning

confidence: 99%

DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis

Ruan¹,

Zhang²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Text-to-image synthesis refers to generating an image from a given text description, the key goal of which lies in photo realism and semantic consistency. Previous methods usually generate an initial image with sentence embedding and then refine it with fine-grained word embedding. Despite the significant progress, the 'aspect' information (e.g., red eyes) contained in the text, referring to several words rather than a word that depicts 'a particular part or feature of something', is often ignored, which is highly helpful for synthesizing image details. How to make better utilization of aspect information in text-to-image synthesis still remains an unresolved challenge. To address this problem, in this paper, we propose a Dynamic Aspect-awarE GAN (DAE-GAN) that represents text information comprehensively from multiple granularities, including sentencelevel, word-level, and aspect-level. Moreover, inspired by human learning behaviors, we develop a novel Aspectaware Dynamic Re-drawer (ADR) for image refinement, in which an Attended Global Refinement (AGR) module and an Aspect-aware Local Refinement (ALR) module are alternately employed. AGR utilizes word-level embedding to globally enhance the previously generated image, while ALR dynamically employs aspect-level embedding to refine image details from a local perspective. Finally, a corresponding matching loss function is designed to ensure the text-image semantic consistency at different levels. Extensive experiments on two well-studied and publicly available datasets (i.e., CUB-200 and COCO) demonstrate the superiority and rationality of our method.

show abstract

“…Soft attention allows the content of an entire scene to be perceived at once, but with varying degrees of importance. A deterministic attention map is predicted and multiplied with the input to suppress unimportant content [77,28]. Soft attention has been used for solving various computer vision tasks such as image classification and segmentation [71,12], image captioning [77], optical character recognition [45], and pose estimation and action recognition [13,47].…”

Section: Literature Reviewmentioning

confidence: 99%

Visual Attention in Imaginative Agents

Rangrej,

Clark

2021

Preprint

View full text Add to dashboard Cite

We present a recurrent agent who perceives surroundings through a series of discrete fixations. At each timestep, the agent imagines a variety of plausible scenes consistent with the fixation history. The next fixation is planned using uncertainty in the content of the imagined scenes. As time progresses, the agent becomes more certain about the content of the surrounding, and the variety in the imagined scenes reduces. The agent is built using a variational autoencoder and normalizing flows, and trained in an unsupervised manner on a proxy task of scene-reconstruction. The latent representations of the imagined scenes are found to be useful for performing pixel-level and scene-level tasks by higher-order modules. The agent is tested on various 2D and 3D datasets.

show abstract

DRAW: A Recurrent Neural Network For Image Generation

Cited by 242 publications

References 0 publications

A Reinforcement Learning Approach for Sequential Spatial Transformer Networks

A Reinforcement Learning Approach for Sequential Spatial Transformer Networks

DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis

Visual Attention in Imaginative Agents

Contact Info

Product

Resources

About