Ensembling with Deep Generative Views

Abstract

Recent generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose, simply by learning from unlabeled image collections. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. Using a pretrained generator, we first find the latent code corresponding to a given real input image. Applying perturbations to the code creates natural variations of the image, which can then be ensembled together at test-time. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars. Critically, we find that several design decisions are required towards making this process work; the perturbation procedure, weighting between the augmentations and original image, and training the classifier on synthesized images can all impact the result. Currently, we find that while test-time ensembling with GAN-based augmentations can offer some small improvements, the remaining bottlenecks are the efficiency and accuracy of the GAN reconstructions, coupled with classifier sensitivities to artifacts in GAN-generated images.

EGAIN: Extended GAn INversion

“…While they are significantly faster, they suffer from inferior image quality [22] [25] [28]. Hybrid methods attempt to balance the trade-offs between the previous two strategies, typically using an encoder only for the initialization of a latent code to start the optimization process [33] [4].…”

A Data Augmentation Pipeline to Generate Synthetic Labeled Datasets of 3D Echocardiography Images Using a GAN

“…This widely common GAN performance drawback is addressed by applying some image post-processing operations [14] on the synthetically generated 3D echocardiography images. In practice synthetic images can be used to train DL models because they represent a good data augmentation strategy [15]. For instance, 3D medical image segmentation is the most common example of a medical task to which DL can turn out to be a good application.…”