Unsupervised Sketch to Photo Synthesis

Liu, Runtao; Yu, Qian; Yu, Stella X.

doi:10.1007/978-3-030-58580-8_3

Cited by 42 publications

(31 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…V shows that FST is helpful for generating diverse anime-faces. [27], which was employed by many translation methods e.g., [8], [23], [24], [47]. The fourth baseline, called "w/o AdaPoLIN w AdaLIN", replaces AdaPoLIN with AdaLIN which is used in UGATIT.…”

Section: E User Studymentioning

confidence: 99%

AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation

Zhu

Wang³

et al. 2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

In this paper, we propose a novel framework to translate a portrait photo-face into an anime appearance. Different from existing translation methods which do not designate specific styles, we aim to synthesize anime-faces which are style-consistent with a given reference anime-face. However, unlike typical translation tasks, such anime-face translation is particularly challenging due to the large and complex variations of appearances among anime-faces. Existing methods often fail to transfer the styles of reference anime-faces to the generated anime-faces, or introduce noticeable artifacts/distortions in the local shapes of their generated anime-faces. We propose a novel GAN-based anime-face translator, called AniGAN, to synthesize high-quality anime-faces. Specifically, a new generator architecture is proposed to simultaneously transfer color/texture styles and transform local facial shapes into anime-like counterparts based on the style of a reference anime-face, while preserving the global structure of the source photo-face. New normalization functions are designed for the generator to further improve local shape transformation and color/texture style transfer. Besides, we propose a double-branch discriminator to learn domain-specific distributions through individual branches and learn cross-domain shared distributions via shared layers, helping generate visually pleasing anime-faces and effectively mitigate artifacts/distortions. Extensive experiments on benchmark datasets qualitatively and quantitatively demonstrate the superiority of our method over state-of-the-art methods.

show abstract

Section: E User Studymentioning

confidence: 99%

AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation

Zhu

Wang³

et al. 2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…Post-processing steps are adopted for small stroke removal, spline fitting [77], and stroke simplification [78]. A few works utilize crowd-sourced free-hand sketches for training [8,9]. They either construct pseudo-paired data by matching sketches and images [8], or propose a method that does not require paired data [9].…”

Section: Sketches and Strokes As Inputmentioning

confidence: 99%

“…A few works utilize crowd-sourced free-hand sketches for training [8,9]. They either construct pseudo-paired data by matching sketches and images [8], or propose a method that does not require paired data [9]. Another aspect of CNN training that has been investigated is the representation of sketches.…”

Section: Sketches and Strokes As Inputmentioning

confidence: 99%

“…As collecting paired training data can be labor intensive, learning from unpaired sketch-photo data in an unsupervised setting is an interesting direction to explore. Liu et al [9] proposed an unsupervised solution by decomposing the synthesis process into a shape translation stage and a content enrichment stage. The shape translation network transforms an input sketch into a gray-scale image, trained using unpaired sketches and images, under the supervision of a cycle-consistency loss.…”

Section: Sketches and Strokes As Inputmentioning

confidence: 99%

“…An interesting next pursuit would be to see if computers can mimic creative processes such as those used by painters in making pictures, or assisting artists or architects in making artistic or architectural designs. In fact, in the past decade, we have witnessed advances in systems that synthesize an image from a text description [1][2][3][4] or from a learned style of content [5], paint a picture given a sketch [6][7][8][9], render a photorealistic scene from a wireframe [10,11], and create virtual reality content from images and videos [12], among others. A comprehensive review of such systems can explain the current state-of-the-art in such pursuits, reveal open challenges, and illuminate future directions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep image synthesis from intuitive user input: A review and perspectives

Guo

Zhang

et al. 2021

Comp. Visual Media

View full text Add to dashboard Cite

In many applications of computer graphics, art, and design, it is desirable for a user to provide intuitive non-image input, such as text, sketch, stroke, graph, or layout, and have a computer system automatically generate photo-realistic images according to that input. While classically, works that allow such automatic image content generation have followed a framework of image retrieval and composition, recent advances in deep generative models such as generative adversarial networks (GANs), variational autoencoders (VAEs), and flow-based methods have enabled more powerful and versatile image generation approaches. This paper reviews recent works for image synthesis given intuitive user input, covering advances in input versatility, image generation methodology, benchmark datasets, and evaluation metrics. This motivates new perspectives on input representation and interactivity, cross fertilization between major image generation paradigms, and evaluation and comparison of generation methods.

show abstract

Geometric Deformation on Objects: Unsupervised Image Manipulation via Conjugation

Cohen

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A novel two-stage approach is proposed for image manipulation and generation. User-interactive image deformation is performed through editing of contours. This is performed in the latent edge space with both color and gradient information. The output of editing is then fed into a multi-scale representation of the image to recover quality output. The model is flexible in terms of transferability and training efficiency.

show abstract

Unsupervised Sketch to Photo Synthesis

Cited by 42 publications

References 29 publications

AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation

AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation

Deep image synthesis from intuitive user input: A review and perspectives

Geometric Deformation on Objects: Unsupervised Image Manipulation via Conjugation

Contact Info

Product

Resources

About