DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation

Abstract

Text-to-image generation aims at generating realistic images which are semantically consistent with the given text. Previous works mainly adopt the multi-stage architecture by stacking generatordiscriminator pairs to engage multiple adversarial training, where the text semantics used to provide generation guidance remain static across all stages. This work argues that text features at each stage should be adaptively re-composed conditioned on the status of the historical stage (i.e., historical stage's text and image features) to provide diversified and accurate semantic guidance during the coarse-to-fine generation process. We thereby propose a novel Dynamical Semantic Evolution GAN (DSE-GAN) to re-compose each stage's text features under a novel single adversarial multi-stage architecture. Specifically, we design (1) Dynamic Semantic Evolution (DSE) module, which first aggregates historical image features to summarize the generative feedback, and then dynamically selects words required to be re-composed at each stage as well as re-composed them by dynamically enhancing or suppressing different granularity subspace's semantics. (2) Single Adversarial Multistage Architecture (SAMA), which extends the previous structure by eliminating complicated multiple adversarial training requirements and therefore allows more stages of text-image interactions, and finally facilitates the DSE module. We conduct comprehensive experiments and show that DSE-GAN achieves 7.48% and 37.8% relative FID improvement on two widely used benchmarks, i.e., respectively.

DreamIdentity: Enhanced Editability for Efficient Face-Identity Preserved Image Generation

“…Early works mainly adopted GAN (Goodfellow et al 2014) as the foundational generative model for this task. Various works have been proposed (Zhang et al 2021;Zhu et al 2019;Xu et al 2018;Zhang et al 2017;Zhang, Xie, and Yang 2018;Liang, Pei, and Lu 2020;Cheng et al 2020;Ruan et al 2021;Tao et al 2020;Li et al 2019;Huang et al 2022) with well-designed textual representations, elegant text-image interactions, and effective loss functions. However, GAN-based models often suffer from training instability and model collapse, making it hard to be trained on largescale datasets (Brock, Donahue, and Simonyan 2018;Kang et al 2023;Schuhmann et al 2021).…”

When Measures are Unreliable: Imperceptible Adversarial Perturbations toward Top-k Multi-Label Learning

“…Then, the top-𝑘 measures, such as P@𝑘, mAP@𝑘, and NDCG@𝑘, will not drop significantly while the specified label has been ranked far behind the top-𝑘 position. In the extreme case, the classifier can never detect certain important categories and produce degraded classification performance, which may result in serious consequences for multimedia applications, such as multimodal emotion recognition [35] and text-to-image generation [20].…”