Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Abstract

Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS), which robustly measures the perceptual fidelity of inpainted images compared to real images via linear separability in a feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation.Code is available at https://github.com/zsyzzsoft/co-mod-gan.

GeoFill: Reference-Based Image Inpainting of Scenes with Complex Geometry

“…ProFill [52] utilizes a contextual attention module and progressively fills the hole with predicted high-confidence pixels. CoModGAN [56] proposes a co-modulation of both conditional and stochastic representations to fill high quality content. Lastly, we compare a two-view SfM based approach [57] -which we refer to as JointDP -by warping the source image with the jointly estimated relative pose and depth using dense correspondence.…”

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

“…To overcome this issue, Zheng et al [48] and Zhao et al [46] propose a VAE-based network that trade-offs between diversity and reconstruction. Zhao et al [47], inspired by the StyleGAN2 [15] modulated convolutions, introduces a comodulation layer for the inpainting task in order to improve both diversity and reconstruction. Noteworthy, a new family of auto-regressive methods [27,34,42], which can handle irregular masks, has recently emerged as a powerful alternative for free-form image inpainting.…”

SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing

“…However, the enforcement of diversity deteriorates the image quality. Inspired by the recent modulation approach [53] for multimodal image inpainting, we propose a similar network architecture specifically for open-domain image editing. The difference is that our modulation layer does not use the features of the input image, which leads to better fidelity and diversity.…”

“…Note that we circumvent direct pixel supervision such as L1 loss [16] for the purpose of encouraging the generation diversity, as suggested in [53]. Some qualitative output results from our trained generator is visualized in Fig.…”

“…Our algorithm surpasses BiCycleGAN and DivCo by a large margin according to FID, mainly due to the benefit of the styleGAN-like structure. And as indicated in [53], the modulation-based conditional generator is intrinsically stochastic w.r.t. the input noise even without explicit diversity constraint used in [24,55].…”