Depth-aware neural style transfer

Abstract

style loss and content loss. However, such pre-trained networks are originally designed for object recognition, and hence the high-level features o en focus on the primary target and neglect other details. As a result, when input images contain multiple objects potentially at di erent depths, the resulting images are o en unsatisfactory because image layout is destroyed and the boundary between the foreground and background as well as di erent objects becomes obscured. We observe that the depth map e ectively re ects the spatial distribution in an image and preserving the depth map of the content image a er stylization helps produce an image that preserves its semantic content. In this paper, we introduce a novel approach for neural style transfer that integrates depth preservation * Corresponding author's email: cmm@nankai.edu.cn Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro t or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permi ed. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci c permission and/or a fee.

Neural Style Transfer: A Review

“…Jiang'17 [89] Fashion Chen'18 [72] 3D 2D Johnson'16 [47] Ulyanov'16 [48] Ulyanov'17 [50] Li'16 [52] Liu'17 [63] Depth Wang'17 [62] Jing'18 [61] Stroke Size Improvement Image Video Huang'17 [78] Gupta'17 [79] Chen'17 [80] Lu'17 [70] Semantic Non-Photorealistic Zhang'17 [87] Photorealistic Azadi'18 [83] Character Figure 2: A taxonomy of NST techniques. Our proposed NST taxonomy extends the IB-AR taxonomy proposed by Kyprianidis et al [14].…”

“…Another limitation of current NST algorithms is that they do not consider the depth information contained in the image. To address this limitation, the depth preserving NST algorithm [63] is proposed. Their approach is to add a depth loss function based on [47] to measure the depth difference between the content image and the stylised image.…”

“…Liu et al [63] Depth Loss Preserving depth maps of content images. Effective at retaining spatial layout and structure of content images, compared with single Gram Loss.…”

Structure-Preserving Neural Style Transfer

“…A preliminary version of this paper has been published in NPAR 2017 [30]. [20], tend to scatter the textures over the stylized outputs, and ignore the structure information in an image.…”

“…Some previous works have demonstrated that preserving structures can produce attractive artistic results [13], [36], [46]. Considering that a depth map can effectively capture the global structure of the scene [16], [40], the early version of this paper [30] integrates depth as an additional representation to preserve overall image layout during style transfer. Which image stylization seems best to you?…”