Sketch-guided Deep Portrait Generation

Abstract

Generating a realistic human class image from a sketch is a unique and challenging problem considering that the human body has a complex structure that must be preserved. Additionally, input sketches often lack important details that are crucial in the generation process, hence making the problem more complicated. In this article, we present an effective method for synthesizing realistic images from human sketches. Our framework incorporates human poses corresponding to locations of key semantic components (e.g., arm, eyes, nose), seeing that its a strong prior for generating human class images. Our sketch-image synthesis framework consists of three stages: semantic keypoint extraction, coarse image generation, and image refinement. First, we extract the semantic keypoints using Part Affinity Fields (PAFs) and a convolutional autoencoder. Then, we integrate the sketch with semantic keypoints to generate a coarse image of a human. Finally, in the image refinement stage, the coarse image is enhanced by a Generative Adversarial Network (GAN) that adopts an architecture carefully designed to avoid checkerboard artifacts and to generate photo-realistic results. We evaluate our method on 6,300 sketch-image pairs and show that our proposed method generates realistic images and compares favorably against state-of-the-art image synthesis methods.

Generative Adversarial Networks–Enabled Human–Artificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends

“… Ho et al (2020) presented a novel sketch-based generation network for full-body images of people. The authors used semantic key points corresponding to essential human body parts as a prior for sketch-image synthesis.…”

“…The system also operates on relatively large images, 512 × 512 pixels, which increases real-world usability, as does the feel of its interface, which is much like any professional painting tool. Ho et al (2020) presented a novel sketch-based generation network for full-body images of people. The authors used semantic key points corresponding to essential human body parts as a prior for sketch-image synthesis.…”

FedSGDCOVID: Federated SGD COVID-19 Detection under Local Differential Privacy Using Chest X-ray Images and Symptom Information

“…CNN not only achieves excellent performance on computer vision tasks such as object detection [ 55 , 56 ], image classification [ 57 ], image generation [ 58 ], tracking task [ 59 ], and face recognition [ 60 ], but also can be used to sequence data [ 61 ]. Some works have achieved great results in COVID-19 detection using CNN architecture on textual data.…”

D3T-GAN: Data-Dependent Domain Transfer GANs for Few-shot Image Generation

“…Recent years have witnessed the development of generative adversarial networks (GANs) [5,10,14,21,22,24,35,39,42,48,50], which learn a function to generate high-dimension images from randomly sampled noises. To be capable of generating realistic images, GANs always rely on a large amount of training samples to fit the real distribution of given categories.…”