Semantic Image Inpainting with Deep Generative Models

Abstract

Semantic image inpainting is a challenging task where large missing regions have to be filled based on the available visual data. Existing methods which extract information from only a single image generally produce unsatisfactory results due to the lack of high level context. In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data. Given a trained generative model, we search for the closest encoding of the corrupted image in the latent image manifold using our context and prior losses. This encoding is then passed through the generative model to infer the missing content. In our method, inference is possible irrespective of how the missing content is structured, while the state-of-the-art learning based method requires specific information about the holes in the training phase. Experiments on three datasets show that our method successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.

Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery

“…To identify anomalies, we learn a model representing normal anatomical variability based on GANs [13]. This method trains a generative model, and a discriminator to distinguish between generated and real data simultaneously (see Figure 2(a)).…”

“…In the spirit of [13], we define a loss function for the mapping of new images to the latent space that comprises two components, a residual loss and a discrimination loss. The residual loss enforces the visual similarity between the generated image G(z γ ) and query image x.…”

“…Discrimination Loss For image inpainting, Yeh et al [13] based the computation of the discrimination loss LD(z γ ) on the discriminator output by feeding the generated image G(z γ ) into the discriminator LD(z γ ) = σ(D (G(z γ )), α), where σ is the sigmoid cross entropy, which defined the discriminator loss of real images during adversarial training, with logits D (G(z γ )) and targets α = 1.…”

“…Radford et al [12] introduced deep convolutional generative adversarial networks (DCGANs) and showed that GANs are capable of capturing semantic image content enabling vector arithmetic for visual concepts. Yeh et al [13] trained GANs on natural images and applied the trained model for semantic image inpainting. Compared to Yeh et al [13], we implement two adaptations for an improved mapping from images to the latent space.…”

“…Yeh et al [13] trained GANs on natural images and applied the trained model for semantic image inpainting. Compared to Yeh et al [13], we implement two adaptations for an improved mapping from images to the latent space. We condition the search in the latent space on the whole query image, and propose a novel variant to guide the search in the latent space (inspired by feature matching [14]).…”