Effective Removal of User-Selected Foreground Object From Facial Images Using a Novel GAN-Based Network

Abstract

This research features a user-friendly method for face de-occlusion in facial images where the user has control of which object to remove. Our system removes one object at a time, however, it is capable of removing multiple objects through repeated application. Although we show the effectiveness of our system on five commonly occurring occluding objects including hands, a medical mask, microphone, sunglasses, and eyeglasses, more types of object can be considered based on the proposed methodology. Our model learns to detect a user-selected, possibly distracting, object in the first stage. Then, the second stage removes the object using the object detection information from the first stage as guidance. To achieve this, we employ GAN-based networks in both stages. Specifically, in the second stage, we integrate both partial and vanilla convolution operations in the generator part of the GAN network. We show that by using this integration, the proposed network can learn a well-incorporated structure and also overcome the problem of visual discrepancies in the affected region of the face. To train our network, we produce a paired synthetic face-occluded dataset. Our model is evaluated using real world images collected from the Internet and publicly available CelebA and CelebA-HQ datasets. Experimental results confirm our model's effectiveness in removing challenging foreground non-face objects from facial images as compared to the existing representative state-of-the-art approaches.

Masked Face Recognition Using Deep Learning: A Review

“…There are few publicly available datasets that include facial image pairs with and without mask objects to sufficiently train the MFR system in a progressing manner. Therefore, this strengthens the requirement of enriching the testbed by additional synthetic images with various types of face masks [29,30], as well as improving the generalization capability of deep learning models. Among the most popular methods used to synthesize the face masks are MaskTheFace [31], MaskedFace-Net [32], deep convolutional neural network (DCNN) [33], CYCLE-GAN [34], Identity Aware Mask GAN (IAMGAN) [35], and starGAN [36].…”

“…It generates a new feature embedding similar to an embedding of an unmasked face of the same identity with unique properties. Din et al [29,30] used a GAN setup with two discriminators to automatically remove the face mask.…”

“…Yu et al [140] proposed a coarse-to-fine GAN-based approach with a novel contextual attention module for image inpainting. Din et al [29,30] used GAN-based image inpainting for image completion through an image-to-image translation approach. Duan et al [141] used GANs to handle the face frontalization and face completion tasks simultaneously.…”

“…The Synthetic Face-Occluded Dataset [30] was created using the publicly available CelebA and CelebA-HQ [150] datasets. CelebA-HQ is a large-scale face attribute dataset with more than 30,000 celebrity images.…”

Haze Relevant Feature Attention Network for Single Image Dehazing

“…Whenever discriminator network notices a difference between two distributions, generator network slightly adjusts its parameters to make the difference disappear, until finally generator network accurately reproduces real data distribution and discriminator network fails to distinguish between true and false. In recent years, a variety of improved GAN has been widely used in the field of image processing, covering almost all the traditional image processing fields, as well as some new applications, such as image editing [17], image translation [18], style transfer [19] and so on.…”