Learning from Simulated and Unsupervised Images through Adversarial Training

Abstract

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Tumor-Aware, Adversarial Domain Adaptation from CT to MRI for Lung Cancer Segmentation

“…Typically used fine-tuning methods require prohibitively large labeled data in the target domain. As an alternative, domain adaptation methods attempt to minimize domain shift either by feature sharing[2] or by learning to reconstruct the target from source domain[3, 4]. In essence, domain adaptation methods learn the marginal distributions [5] to transform source to target domain.…”

Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement

“…This can be handled by using paired source-target data to regularize the translation [9]. In the unpaired setting, prior work has constrained the translated images to be close to the source images [30] but this only works for small domain shifts. Most current methods use a combination of task-specific losses (i.e., preserving the task network's output after translation) [25], image-space and feature-space adversarial losses, cycleconsistency losses, and semantic losses (i.e., preserving the semantics of the image after translation) [14,26,42].…”

“…While our focus is on improving visual quality, our method can be used for domain adaptation. Deep networks can be trained on large-scale, labeled synthetic datasets [28,41] and prior work has looked at adapting them to improve their performance on real data [9,30,25]. Many of these methods impose a task-specific loss on this adaptation [14,26,42].…”

Realistic in silico generation and augmentation of single cell RNA-seq data using Generative Adversarial Neural Networks

“…While the number of biological samples might be limited, realistic in silico generation of observations could accommodate for this unfavorable setting. In practice, in silico generation has seen success in computer vision when used for 'data augmentation', whereby in silico generated samples are used alongside the original ones to artificially increase the number of observations 3 . While classically realistic data modeling relies on a thorough understanding of the laws underlying the production of such data, current methods of choice for photo-realistic image generation rely on Deep Learning-based Generative Adversarial Networks (GANs) [4][5][6][7] and Variational Autoencoders (VAEs) 8,9 .…”