Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence

Lee, Junsoo; Kim, Eungyeup; Lee, Yunsung; Kim, Dong‐Jun; Chang, Joonho; Choo, Jaegul

doi:10.1109/cvpr42600.2020.00584

Cited by 153 publications

(145 citation statements)

References 28 publications

Supporting

Mentioning

144

Contrasting

Order By: Relevance

“…More recently, Ref. [27] proposed a reference-based sketch colorization method based on a transformationaware attention module. However, this method still shares the problem of color style inconsistency between the reference and the output.…”

Section: Deep Conditional Image-to-image Synthesismentioning

confidence: 99%

Reference-guided structure-aware deep sketch colorization for cartoons

Liu

et al. 2021

Comp. Visual Media

View full text Add to dashboard Cite

Digital cartoon production requires extensive manual labor to colorize sketches with visually pleasant color composition and color shading. During colorization, the artist usually takes an existing cartoon image as color guidance, particularly when colorizing related characters or an animation sequence. Reference-guided colorization is more intuitive than colorization with other hints, such as color points or scribbles, or text-based hints. Unfortunately, reference-guided colorization is challenging since the style of the colorized image should match the style of the reference image in terms of both global color composition and local color shading. In this paper, we propose a novel learning-based framework which colorizes a sketch based on a color style feature extracted from a reference color image. Our framework contains a color style extractor to extract the color feature from a color image, a colorization network to generate multi-scale output images by combining a sketch and a color feature, and a multi-scale discriminator to improve the reality of the output image. Extensive qualitative and quantitative evaluations show that our method outperforms existing methods, providing both superior visual quality and style reference consistency in the task of reference-based colorization.

show abstract

Section: Deep Conditional Image-to-image Synthesismentioning

confidence: 99%

Reference-guided structure-aware deep sketch colorization for cartoons

Liu

et al. 2021

Comp. Visual Media

View full text Add to dashboard Cite

show abstract

“…The work of [Gonzalez-Garcia et al 2018] decouples images into a shared part and an exclusive part to achieve multi-modal image translation. Furthermore, several works (e.g., [Lee et al 2020c;Yu et al 2019a]) have extended the disentangled representation to provide domain-invariant and domain-specific representations to perform multi-domain and multi-modal translations simultaneously. However, these methods have focused on multi-domain translation and their disentanglement mainly aims at holistic attributes whereas our method focuses on disentangled representations that respect facial component structure and support detailed editing.…”

Section: Neural Image Disentanglementmentioning

confidence: 99%

DeepFaceEditing

et al. 2021

View full text Add to dashboard Cite

Recent facial image synthesis methods have been mainly based on conditional generative models. Sketch-based conditions can effectively describe the geometry of faces, including the contours of facial components, hair structures, as well as salient edges (e.g., wrinkles) on face surfaces but lack effective control of appearance, which is influenced by color, material, lighting condition, etc. To have more control of generated results, one possible approach is to apply existing disentangling works to disentangle face images into geometry and appearance representations. However, existing disentangling methods are not optimized for human face editing, and cannot achieve fine control of facial details such as wrinkles. To address this issue, we propose DeepFaceEditing, a structured disentanglement framework specifically designed for face images to support face generation and editing with disentangled control of geometry and appearance. We adopt a local-to-global approach to incorporate the face domain knowledge: local component images are decomposed into geometry and appearance representations, which are fused consistently using a global fusion module to improve generation quality. We exploit sketches to assist in extracting a better geometry representation, which also supports intuitive geometry editing via sketching. The resulting method can either extract the geometry and appearance representations from face images, or directly extract the geometry representation from face sketches. Such representations allow users to easily edit and synthesize face images, with decoupled control of their geometry and appearance. Both qualitative and quantitative evaluations show the superior detail and appearance control abilities of our method compared to state-of-the-art methods.

show abstract

“…Colorization is recent active research yet a difficult subject in the realm of image processing, with the goal of quickly predicting and colorizing grayscale images by analyzing image content with a computer. Existing colorizing algorithms can be classified into three categories depending on the information provided by humans: scribble-based [5][6][7][8][9][10], example-based [11][12][13][14][15], and learning-based [1][2][3][4][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32] methods.…”

Section: Introductionmentioning

confidence: 99%

“…generate a colorized image. [15] designed a spatially corresponding feature transfer module that uses self-attention to learn the relationship between the input sketch image and the reference image. Although scribble-based and examplebased methods can save a large amount of time when compared to all-manual colorizing, each processed image must still be assisted by providing relevant color information.…”

Section: Introductionmentioning

confidence: 99%

Two-stage pyramidal convolutional neural networks for image colorization

Wei

Kuo

et al. 2021

SIP

View full text Add to dashboard Cite

The development of colorization algorithms through deep learning has become the current research trend. These algorithms colorize grayscale images automatically and quickly, but the colors produced are usually subdued and have low saturation. This research addresses this issue of existing algorithms by presenting a two-stage convolutional neural network (CNN) structure with the first and second stages being a chroma map generation network and a refinement network, respectively. To begin, we convert the color space of an image from RGB to HSV to predict its low-resolution chroma components and therefore reduce the computational complexity. Following that, the first-stage output is zoomed in and its detail is enhanced with a pyramidal CNN, resulting in a colorized image. Experiments show that, while using fewer parameters, our methodology produces results with more realistic color and higher saturation than existing methods.

show abstract

Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence

Cited by 153 publications

References 28 publications

Reference-guided structure-aware deep sketch colorization for cartoons

Reference-guided structure-aware deep sketch colorization for cartoons

DeepFaceEditing

Two-stage pyramidal convolutional neural networks for image colorization

Contact Info

Product

Resources

About