An Unsupervised Way to Understand Artifact Generating Internal Units in Generative Neural Networks

Abstract

Despite significant improvements on the image generation performance of Generative Adversarial Networks (GANs), generations with low visual fidelity still have been observed. As widely used metrics for GANs focus more on the overall performance of the model, evaluation on the quality of individual generations or detection of defective generations is challenging. While recent studies try to detect featuremap units that cause artifacts and evaluate individual samples, these approaches require additional resources such as external networks or a number of training data to approximate the real data manifold. In this work, we propose the concept of local activation, and devise a metric on the local activation to detect artifact generations without additional supervision. We empirically verify that our approach can detect and correct artifact generations from GANs with various datasets. Finally, we discuss a geometrical analysis to partially reveal the relation between the proposed concept and low visual fidelity.

DiffGAR: Model-Agnostic Restoration from Generative Artifacts Using Image-to-Image Diffusion Models

“…Concurrently, the diffusion models have emerged as a promising generative modeling framework, advancing the development of image, audio, and video generation tasks. However, artifacts are often generated due to architectural limitations of the generative model itself and its inability to capture the complete data manifold pattern [47], [14], [39], highlighting the importance of generative artifacts removal for producing more visually appealing images.…”

“…Recently, there are a few works focusing on exploring the unique artifacts in GAN model architectures. The author [14] removes artifacts through ablating units that are related to artifact generations.…”

“…As far as we know, only a few works have been proposed to deal with the artifacts of generative models. Two works [14], [40] have been conducted to detect artifacts produced by GAN models. However, these models are model-specific since they are limited to the latent space of GAN.…”