GANILLA: Generative adversarial networks for image to illustration translation

Abstract

In this paper, we explore illustrations in children's books as a new domain in unpaired image-to-image translation. We show that although the current state-of-the-art image-to-image translation models successfully transfer either the style or the content, they fail to transfer both at the same time. We propose a new generator network to address this issue and show that the resulting network strikes a better balance between style and content.There are no well-defined or agreed-upon evaluation metrics for unpaired imageto-image translation. So far, the success of image translation models has been based on subjective, qualitative visual comparison on a limited number of images. To address this problem, we propose a new framework for the quantitative evaluation of image-toillustration models, where both content and style are taken into account using separate classifiers. In this new evaluation framework, our proposed model performs better than the current state-of-the-art models on the illustrations dataset. Our code and pretrained models can be found at https://github.com/giddyyupp/ganilla.

Pseudo-Supervised Learning for Semantic Multi-Style Transfer

“…The animation images are also segmented manually according to the same semantic classes of the photo images. The previous models [1], [2], [20], [21], [36]- [38] were retrained with these datasets for fair comparisons. All images were resized to 256 × 256 for training.…”

“…Fig. 7 compares the qualitative results of our method with the CycleGAN [1], DiscoGAN [21], DualGAN [20], MUNIT [36], CartoonGAN [2], U-GAT-IT [37], and GANILLA [38]. It can be seen by comparing the results object-by-object that our method produces the best results with the fine details of the target anime object styles.…”

“…Also, because the generators of the previous methods fail to learn the dedicated mapping between the tree in the real photo images and that in the target style images, the unique texture of the tree in the target style domain is lost in the resulting outputs. The qualitative results show that our proposed pseudo-supervised learning framework can [20], (e) MUNIT [36], (f) CartoonGAN [2], (g) U-GAT-IT [37], (h) GANILLA [38], and (i) our method. (j) Real anime sample from the target anime for visual comparison.…”

Deep Generative Adversarial Networks for Image-to-Image Translation: A Review

“…Chen et al [116] propose an adversarial gated networks, called Gated-GAN, to transfer multiple styles while using a single model based on three modalities: an encoder, a gated transformer, and a decoder. GANILLA [117] is a proposed novel framework with the ability to better balance between content and style. Style transfer is the process of rendering the content of an image with a specific style while preserving the content, as shown in Figure 6.…”

“…Metrics of success of image-to-image translation usually evaluate the quality of generated images while using a limited number of test images or user studies. The evaluation of a limited number of test images must consider both style and content simultaneously, which is difficult to do [117]. In addition, user studies are based on human judgment, which is a subjective metric [1].…”