Detecting Overfitting of Deep Generative Networks via Latent Recovery

Abstract

State of the art deep generative networks are capable of producing images with such incredible realism that they can be suspected of memorizing training images. It is why it is not uncommon to include visualizations of training set nearest neighbors, to suggest generated images are not simply memorized. We demonstrate this is not sufficient and motivates the need to study memorization/overfitting of deep generators with more scrutiny. This paper addresses this question by i) showing how simple losses are highly effective at reconstructing images for deep generators ii) analyzing the statistics of reconstruction errors when reconstructing training and validation images, which is the standard way to analyze overfitting in machine learning. Using this methodology, this paper shows that overfitting is not detectable in the pure GAN models proposed in the literature, in contrast with those using hybrid adversarial losses, which are amongst the most widely applied generative methods. The paper also shows that standard GAN evaluation metrics fail to capture memorization for some deep generators. Finally, the paper also shows how off-theshelf GAN generators can be successfully applied to face inpainting and face super-resolution using the proposed reconstruction method, without hybrid adversarial losses.

Determining overfitting and underfitting in generative adversarial networks using Fréchet distance

“…Considering the role played by ImageNet Large Scale Visual Recognition Challenge [12] (ILSVRC) at the advancement of CNNs for image classification models, the importance of an evaluation metric can be understood better. In this respect, many different evaluation metrics have been proposed so far by the researchers [9], including Recovery Error [13], IS [5], FID [6], and Kernel Inception Distance (KID) [14]. Although the IS and FID gained gradually increasing reputations and are commonly used, unfortunately both of them fail when it comes to determine whether the proposed GAN model is overfitting or underfitting.…”

“…In order to detect overfitting, in [13], using recovery error was proposed. By optimizing the random input vector z , a GAN model tries to generate mimic of each image in the training set and validation set, and obtains recovery error distributions between training set-generated images (mimic of training set) and validation setgenerated images (mimic of validation set).…”

“…Although FID takes into account the real data set and its statistics and became a standard [13] with these properties for GAN evaluation metric, since the FID is one-dimensional real value, by just looking at FID value, we cannot evaluate whether the GAN model is overfit or underfit.…”

“…In order to address this issue, we utilized the observation that the derivative of FD indicates whether the GAN model is overfitting or underfitting. Furthermore, in contrast to [13], for calculating the FD, only forward propagation, through the trained discriminator of the GAN, is needed which makes this method computationally more feasible for determining overfitting and underfitting.…”

OEBR-GAN: Object Extraction and Background Recovery Generative Adversarial Networks

“…Our model has been implemented on TensorFlow 1.14.0 version, CUDA Toolkit version 10. [36]. MRE defines the distribution of error into a single value.…”