Multi-Density Sketch-to-Image Translation Network

Abstract

Sketch-to-image (S2I) translation plays an important role in image synthesis and manipulation tasks, such as photo editing and colorization. Some specific S2I translation including sketch-to-photo and sketch-to-painting can be used as powerful tools in the art design industry. However, previous methods only support S2I translation with a single level of density, which gives less flexibility to users for controlling the input sketches. In this work, we propose the first multi-level density sketch-to-image translation framework, which allows the input sketch to cover a wide range from rough object outlines to micro structures. Moreover, to tackle the problem of noncontinuous representation of multi-level density input sketches, we project the density level into a continuous latent space, which can then be linearly controlled by a parameter. This allows users to conveniently control the densities of input sketches and generation of images. Moreover, our method has been successfully verified on various datasets for different applications including face editing, multi-modal sketchto-photo translation, and anime colorization, providing coarse-tofine levels of controls to these applications.

Deep Learning for Free-Hand Sketch: A Survey

“…Sketch and photo based mutual generation (translation/synthesis) is a classic cross-modal topic of sketch research covering both: (i) sketch-to-photo generation [17], [273], [274], (ii) photo-to-sketch generation [55], [271], [272], [275]. In particular, sketch-to-photo generation methods have addressed: (a) sketch to photo [267], (b) sketch & photo to photo [22], [263], (c) sketch/edge & color to photo [58].…”

Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network

“…However, this type of style control is not accurate, and it can be applied only to images in a global way. Later works [47], [33], [15] used an example to semantically control the output images, while other works [56], [60], [16] tried to locally control attributes of the output such as color and pose, based on users' input. Most recently, motivated by the high quality of StyleGAN [22], some works begin to incorporate it into the I2I framework.…”