AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks

Abstract

Generative Adversarial Networks (GANs) are formulated as minimax game problems, whereby generators attempt to approach real data distributions by virtue of adversarial learning against discriminators. The intrinsic problem complexity poses the challenge to enhance the performance of generative networks. In this work, we aim to boost model learning from the perspective of network architectures, by incorporating recent progress on automated architecture search into GANs. To this end, we propose a fully differentiable search framework for generative adversarial networks, dubbed alphaGAN. The searching process is formalized as solving a bi-level minimax optimization problem, in which the outer-level objective aims for seeking a suitable network architecture towards pure Nash Equilibrium conditioned on the generator and the discriminator network parameters optimized with a traditional GAN loss in the inner level. The entire optimization performs a first-order method by alternately minimizing the two-level objective in a fully differentiable manner, enabling architecture search to be completed in an enormous search space. Extensive experiments on CIFAR-10 and STL-10 datasets show that our algorithm can obtain high-performing architectures only with 3-GPU hours on a single GPU in the search space comprised of approximate 2 × 10 11 possible configurations. We also provide a comprehensive analysis on the behavior of the searching process and the properties of searched architectures, which would benefit further research on architectures for generative models. Pretrained models and codes are available at https://github.com/yuesongtian/AlphaGAN.

EGANS: Evolutionary Generative Adversarial Network Search for Zero-Shot Learning

“…Since GAs can tackle difficult problems that cannot be directly solved, many GA-based algorithms for the NAS have emerged recently [74]- [77]. Although some NAS methods have been introduced to automatically find network architectures for generative models (e.g., GANs) [41], [43], [45], [46], ENAS-based GANs lack of enough attention [47]. Ying [42] and Liu [61] initially introduce an evolutionary algorithmbased NAS framework to search GANs stably, but the models are based on the unconditional image generation task.…”

“…Based on the gradient optimization, Gao et al [44] proposed the AdversarialNAS to search generator and discriminator simultaneously with an adversarial loss function. Meanwhile, Tian et al [45] introduced a fully differentiable search framework for the searching generator and discriminator of GAN. Since the architectures of the generator and discriminator in AdversarialNAS are deeply coupled, the search complexity and the instability of GAN training are increased.…”

Neural Architecture Search Survey: A Computer Vision Perspective

“…Therefore, the developed method was found to be very efficient in finding superior generative models in the proposed large search space, proving its performance and superiority. In a 2021 study, Tian et al [ 117 ] proposed a fully differentiable search framework, also known as alphaGAN. The search process was built as equations to solve bi-level minimax optimization problems.…”

A comprehensive review of generative adversarial networks: Fundamentals, applications, and challenges

“…There are various metrics used to measure the divergence between the probability distributions, including Jensen–Shannon (JS) divergence (Goodfellow et al, 2014), Wasserstein distance (Arjovsky et al, 2017), and maximum mean discrepancy (MMD; Huang et al, 2021). The diversity of the objective functions leads to several extensions of GANs like Wasserstein GAN (WGAN; Arjovsky et al, 2017), LSGAN (Mao et al, 2017), Cramér GAN (Bellemare et al, 2017), f‐GAN (Nowozin et al, 2016), AlphaGAN (Y. Tian et al, 2020), and ‐GAN (Gnanha et al, 2022). On the other side, different validation metrics are used to judge the quality and diversity of the generated synthetic data.…”