Improving Complex Scene Generation by Enhancing Multi-Scale Representations of GAN Discriminators

Abstract

While recent advances of GAN models enabled photo-realistic synthesis of various object images, challenges still remain in modeling more complex image distributions such as scenes with multiple objects. The difficulty lies in the high structural complexity of scene images, where the discriminator carries a heavy burden in discriminating complex structural differences between real and fake scene images. Therefore, enhancing the discriminative capability of the discriminator could be one of the effective strategies to improve the generation performance of GAN models. In this paper, we explore ways to boost the discriminative capability by leveraging two recent paradigms on visual representation learning: selfsupervised learning and transfer learning. As the first approach, we propose a self-supervised auxiliary task tailored to enhance the multi-scale representations of the discriminator. In the second approach, we further enhance the discriminator by utilizing pretrained representations from various scene understanding models. To fully utilize knowledge from multiple expert models, we propose a multi-scale feature ensemble to mix multi-sale representations. Empirical results on challenging scene datasets demonstrate that the proposed strategies significantly advance the generation performance, enabling diverse and photo-realistic synthesis of complex scene images. a a This is an extended and revised version of a conference paper [1] that was presented in WACV 2023. In this paper, we introduce an additional approach that further improves the discriminator representations by utilizing pretrained expert models (Section V). We validate the improvement with additional experimental results and a corresponding ablation study (Section VI).