Dual Attention GANs for Semantic Image Synthesis

Abstract

We propose a novel edge guided generative adversarial network with contrastive learning (ECGAN) for the challenging semantic image synthesis task. Although considerable improvements have been achieved by the community in the recent period, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures; 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects); 3) Existing semantic image synthesis methods focus on modeling "local" semantic information from a single input semantic layout. However, they ignore "global" semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use the edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. We further propose a novel multi-scale contrastive learning method that aims to push same-class features from different scales closer together being able to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts from different scales. Experiments on three challenging datasets show that our methods achieve significantly better results than state-of-the-art approaches. The source code is available at https://github.com/Ha0Tang/ECGAN.

AniFormer: Data-driven 3D Animation with Transformer

“…After summing with the temporal embedding E t ∈ R T ×C×V , the resulting feature vector Z is sent to the AniFormer encoder. In our case, original batch normalization layers are replaced with Instance Normalization [29] layers to preserve the instance style [15,19,27,28]. AniFormer Encoder.…”

Cross-View Exocentric to Egocentric Video Synthesis

“…Generative Adversarial Networks (GANs). Over the last few years, GANs [16] have been shown effectively in many image generation and translation tasks [18,24,37,[39][40][41][42][43][44]57]. For example, Isola et al [18] propose Pix2Pix adversarial learning framework for paired image generation.…”

Diverse Semantic Image Synthesis via Probability Distribution Modeling

“…Pix2pixHD [42] improves Pix2Pix by proposing coarseto-fine generator and discriminators. Subsequent meth-ods [32,27,39,46,37,51] further explore how to synthesize high quality images from semantic masks and achieve significant improvements. Besides using class-level semantic masks, some works also consider instance-level information for image synthesis, since the semantic mask itself does not provide sufficient information to synthesize instances especially in complex environments with multiple of them interacting with each other.…”