AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style

Фролов, С. А.; Sharma, Avneesh; Hees, J.J. van; Karayil, Tushar; Raue, Federico; Dengel, Andreas

doi:10.48550/arxiv.2103.13722

Cited by 2 publications

(2 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Visual guidance has attracted broad attention in image synthesis and editing thanks to its wide applications. Typically, visual guidance represents certain image properties in pixel space such as segmentation maps [2], [3], keypoints [55]- [57], rendered geometry [58]- [63], edge or sketchy maps [64]- [66], and scene layouts [67]- [71] as illustrated in Fig. 1.…”

Section: Visual Guidancementioning

confidence: 99%

Expired

Zhan¹

2022

Preprint

View full text Add to dashboard Cite

As information exists in various modalities in real world, effective interaction and fusion among multimodal information plays a key role for the creation and perception of multimodal data in computer vision and deep learning research. With superb power in modelling the interaction among multimodal information, multimodal image synthesis and editing have become a hot research topic in recent years. Different from traditional visual guidance which provides explicit clues, multimodal guidance offers intuitive and flexible means in image synthesis and editing. On the other hand, this field is also facing several challenges in alignment of features with inherent modality gaps, synthesis of high-resolution images, faithful evaluation metrics, etc. In this survey, we comprehensively contextualize the advance of the recent multimodal image synthesis \& editing and formulate taxonomies according to data modality and model architectures. We start with an introduction to different types of guidance modalities in image synthesis and editing. We then describe multimodal image synthesis and editing approaches extensively with detailed frameworks including Generative Adversarial Networks (GANs), GAN Inversion, Transformers, and other methods such as NeRF and Diffusion models. This is followed by a comprehensive description of benchmark datasets and corresponding evaluation metrics as widely adopted in multimodal image synthesis and editing, as well as detailed comparisons of different synthesis methods with analysis of respective advantages and limitations. Finally, we provide insights into the current research challenges and possible future research directions. We hope this survey could lay a sound and valuable foundation for future development of multimodal image synthesis and editing. A project associated with this survey is available at \href{https://github.com/fnzhan/MISE}{https://github.com/fnzhan/MISE}.

show abstract

Section: Visual Guidancementioning

confidence: 99%

Expired

Zhan¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It consists of a generator network that learns to produce realistic images, and a discriminator network that seeks to discern between real and generated images. This framework has since successfully been applied to numerous applications such as (unconditional and conditional) image synthesis [21,2,13], image editing [26], text-to-image translation [35,12], image-to-image translation [19], image super-resolution [25], and representation learning [32]. Given the breakthroughs of deep learning enabled by convolutional neural networks (CNNs), GANs typically consist of CNN layers.…”

Section: Introductionmentioning

confidence: 99%

Combining Transformer Generators with Convolutional Discriminators

Durall¹,

Фролов²,

Hees³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Transformer models have recently attracted much interest from computer vision researchers and have since been successfully employed for several problems traditionally addressed with convolutional neural networks. At the same time, image synthesis using generative adversarial networks (GANs) has drastically improved over the last few years. The recently proposed TransGAN is the first GAN using only transformer-based architectures and achieves competitive results when compared to convolutional GANs. However, since transformers are datahungry architectures, TransGAN requires data augmentation, an auxiliary super-resolution task during training, and a masking prior to guide the self-attention mechanism. In this paper, we study the combination of a transformer-based generator and convolutional discriminator and successfully remove the need of the aforementioned required design choices. We evaluate our approach by conducting a benchmark of well-known CNN discriminators, ablate the size of the transformer-based generator, and show that combining both architectural elements into a hybrid model leads to better results. Furthermore, we investigate the frequency spectrum properties of generated images and observe that our model retains the benefits of an attention based generator.

show abstract

AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style

Cited by 2 publications

References 35 publications

Expired

Expired

Combining Transformer Generators with Convolutional Discriminators

Contact Info

Product

Resources

About