Efficient Generation of Multiple Sketch Styles Using a Single Network

Bae, Seho; Din, Nizam Ud; Javed, Kamran; Yi, Juneho

doi:10.1109/access.2019.2931544

Cited by 8 publications

(6 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GAN has a proven powerful solution to generate natural looking results [18]. Due to its success for various tasks, GANs are widely used for problems such as domain translation [19][20][21], texture synthesis [22,23], image inpainting [7,8,11,12,21,[24][25][26] and shadow removal [6,[27][28][29][30].…”

Section: Related Workmentioning

confidence: 99%

Throwaway Shadows Using Parallel Encoders Generative Adversarial Network

et al. 2022

Self Cite

View full text Add to dashboard Cite

Face photographs taken on a bright sunny day or in floodlight contain unnecessary shadows of objects on the face. Most previous works deal with removing shadow from scene images and struggle with doing so for facial images. Faces have a complex semantic structure, due to which shadow removal is challenging. The aim of this research is to remove the shadow of an object in facial images. We propose a novel generative adversarial network (GAN) based image-to-image translation approach for shadow removal in face images. The first stage of our model automatically produces a binary segmentation mask for the shadow region. Then, the second stage, which is a GAN-based network, removes the object shadow and synthesizes the effected region. The generator network of our GAN has two parallel encoders—one is standard convolution path and the other is a partial convolution. We find that this combination in the generator results not only in learning an incorporated semantic structure but also in disentangling visual discrepancies problems under the shadow area. In addition to GAN loss, we exploit low level L1, structural level SSIM and perceptual loss from a pre-trained loss network for better texture and perceptual quality, respectively. Since there is no paired dataset for the shadow removal problem, we created a synthetic shadow dataset for training our network in a supervised manner. The proposed approach effectively removes shadows from real and synthetic test samples, while retaining complex facial semantics. Experimental evaluations consistently show the advantages of the proposed method over several representative state-of-the-art approaches.

show abstract

Section: Related Workmentioning

confidence: 99%

Throwaway Shadows Using Parallel Encoders Generative Adversarial Network

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…A symmetric shape allows a network to have a large number of feature maps in the expansive path, facilitating the transfer of more information. Due to its simple architecture and better performance, other work has also exploited the U-Net architecture with minor modifications for image inpainting [17], multiple sketch styles generation [37], image unmosaicing [11], [38], object removal [9] and object detection in facial images [16]. Nizam et al [16] employed a simple U-Net architecture to efficiently detect medical masks in facial images.…”

Section: Related Workmentioning

confidence: 99%

“…Nizam et al [16] employed a simple U-Net architecture to efficiently detect medical masks in facial images. Some models [9], [11], [37], [38] improved performance by using a discriminator along with the U-Net architecture. We also use the U-Net architecture along with a discriminator and target object label encoder to detect multiple types of nonface objects in facial images.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Effective Removal of User-Selected Foreground Object From Facial Images Using a Novel GAN-Based Network

et al. 2020

Self Cite

View full text Add to dashboard Cite

This research features a user-friendly method for face de-occlusion in facial images where the user has control of which object to remove. Our system removes one object at a time, however, it is capable of removing multiple objects through repeated application. Although we show the effectiveness of our system on five commonly occurring occluding objects including hands, a medical mask, microphone, sunglasses, and eyeglasses, more types of object can be considered based on the proposed methodology. Our model learns to detect a user-selected, possibly distracting, object in the first stage. Then, the second stage removes the object using the object detection information from the first stage as guidance. To achieve this, we employ GAN-based networks in both stages. Specifically, in the second stage, we integrate both partial and vanilla convolution operations in the generator part of the GAN network. We show that by using this integration, the proposed network can learn a well-incorporated structure and also overcome the problem of visual discrepancies in the affected region of the face. To train our network, we produce a paired synthetic face-occluded dataset. Our model is evaluated using real world images collected from the Internet and publicly available CelebA and CelebA-HQ datasets. Experimental results confirm our model's effectiveness in removing challenging foreground non-face objects from facial images as compared to the existing representative state-of-the-art approaches.

show abstract

“…GAN has shown promising ability to produce natural looking outputs [19]. Due to this ability, GAN's have extensively been used for task such as texture synthesis [20,21], domain translation [22][23][24] and image inpainting [9][10][11][15][16][17]24,25].…”

Section: Related Workmentioning

confidence: 99%

Interactive Removal of Microphone Object in Facial Images

et al. 2019

Self Cite

View full text Add to dashboard Cite

Removing a specific object from an image and replacing the hole left behind with visually plausible backgrounds is a very intriguing task. While recent deep learning based object removal methods have shown promising results on this task for some structured scenes, none of them have addressed the problem of object removal in facial images. The objective of this work is to remove microphone object in facial images and fill hole with correct facial semantics and fine details. To make our solution practically useful, we present an interactive method called MRGAN, where the user roughly provides the microphone region. For filling the hole, we employ a Generative Adversarial Network based image-to-image translation approach. We break the problem into two stages: inpainter and refiner. The inpainter estimates coarse prediction by roughly filling in the microphone region followed by the refiner which produces fine details under the microphone region. We unite perceptual loss, reconstruction loss and adversarial loss as joint loss function for generating a realistic face and similar structure to the ground truth. Because facial image pairs with or without microphone do not exist, we have trained our method on a synthetically generated microphone dataset from CelebA face images and evaluated on real world microphone images. Our extensive evaluation shows that MRGAN performs better than state-of-the-art image manipulation methods on real microphone images although we only train our method using the synthetic dataset created. Additionally, we provide ablation studies for the integrated loss function and for different network arrangements.

show abstract

Efficient Generation of Multiple Sketch Styles Using a Single Network

Cited by 8 publications

References 28 publications

Throwaway Shadows Using Parallel Encoders Generative Adversarial Network

Throwaway Shadows Using Parallel Encoders Generative Adversarial Network

Effective Removal of User-Selected Foreground Object From Facial Images Using a Novel GAN-Based Network

Interactive Removal of Microphone Object in Facial Images

Contact Info

Product

Resources

About