StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis

Abstract

Generative Adversarial Network (GAN) is one of the state-of-the-art generative models for realistic image synthesis. While training and evaluating GAN becomes increasingly important, the current GAN research ecosystem does not provide reliable benchmarks for which the evaluation is conducted consistently and fairly. Furthermore, because there are few validated GAN implementations, researchers devote considerable time to reproducing baselines. We study the taxonomy of GAN approaches and present a new open-source library named StudioGAN. StudioGAN supports 7 GAN architectures, 9 conditioning methods, 4 adversarial losses, 13 regularization modules, 3 differentiable augmentations, 7 evaluation metrics, and 5 evaluation backbones. With our training and evaluation protocol, we present a large-scale benchmark using various datasets (CIFAR10, ImageNet, AFHQv2, FFHQ, and Baby/Papa/Granpa-ImageNet) and 3 different evaluation backbones (InceptionV3, SwAV, and Swin Transformer). Unlike other benchmarks used in the GAN community, we train representative GANs, including BigGAN, StyleGAN2, and StyleGAN3, in a unified training pipeline and quantify generation performance with 7 evaluation metrics. The benchmark evaluates other cutting-edge generative models (e.g., StyleGAN-XL, ADM, MaskGIT, and RQ-Transformer). StudioGAN provides GAN implementations, training, and evaluation scripts with the pre-trained weights. StudioGAN is available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.

Extracting Training Data from Diffusion Models

“…We study three GAN architectures, all implemented using the StudioGAN framework [42]: BigGAN [6], MHGAN [74], and StyleGAN [44]. Figure 14 shows the membership inference results.…”

medigan: a Python library of pretrained generative models for medical image synthesis

“…Furthermore, the FID has been observed to vary based on the input image resizing methods and ImageNet backbone feature extraction model types. 31 Based on this, we further hypothesize a susceptibility of the FID to variation due to (a) different backbone feature extractor weights and random seed initializations, (b) different medical and nonmedical backbone model pretraining datasets, (c) different image normalization procedures for real and synthetic dataset, (d) nuances between different frameworks and libraries used for FID calculation, and (f) the dataset sizes used to compute the FID. Such variations can obstruct a reliable comparison of synthetic images generated by different generative models.…”

“…A plethora of different GAN network architectures has been proposed 7,31 starting with a deep convolutional GAN (DCGAN) 32 neural network architecture of both D and G. Later approaches, e.g., include a ResNet-based architecture as backbone 29 and progressively-grow the generator and discriminator networks during training to enable high-resolution image synthesis (PGGAN). 33 Another line of research has been focusing on conditioning the output of GANs based on discrete or continuous labels.…”

“…42,43 However, this approach is manual, laborious and costly, and, hence, research attention has been devoted to automating generative model evaluation, 44,45 including: i. Metrics for automated analysis of the synthetic data and its distribution, such as the inception score (IS) 17 and Fréchet inception distance (FID). 46 Both metrics are popular in computer vision, 31 whereas the latter also has seen widespread adoption in medical imaging. 7 FID is based on a pretrained Inception 47 model (e.g., v1, 48 v3 47 ) to extract features from synthetic and real datasets, which are then fitted to multivariate Gaussians X (e.g., real) and Y (e.g., synthetic) with means μ X and μ Y and covariance matrices Σ X and Σ Y .…”

“…Related libraries, such as pygan, 53 torchGAN, 54 vegans, 55 imaginaire, 56 TF-GAN, 57 PyTorch-GAN, 58 keras-GAN, 59 mimicry, 60 and studioGAN, 31 have focused on facilitating the implementation, training, and comparative evaluation of GANs in computer vision (CV). Despite a strong focus on language models, the HuggingFace transformers library and model hub 61 also contain a few pretrained computer vision GAN models.…”