Deep Photo Style Transfer

Abstract

Figure 1: Given a reference style image (a) and an input image (b), we seek to create an output image of the same scene as the input, but with the style of the reference image. The Neural Style algorithm [5] (c) successfully transfers colors, but also introduces distortions that make the output look like a painting, which is undesirable in the context of photo style transfer. In comparison, our result (d) transfers the color of the reference style image equally well while preserving the photorealism of the output. On the right (e), we show 3 insets of (b), (c), and (d) (in that order). Zoom in to compare results. AbstractThis paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style. Our approach builds upon the recent work on painterly transfer that separates style from the content of an image by considering different layers of a neural network. However, as is, this approach is not suitable for photorealistic style transfer. Even when both the input and reference images are photographs, the output still exhibits distortions reminiscent of a painting. Our contribution is to constrain the transformation from the input to the output to be locally affine in colorspace, and to express this constraint as a custom fully differentiable energy term. We show that this approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, including transfer of the time of day, weather, season, and artistic edits.

Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow

“…Despite being a research topic for a long time [RHW88], the interest in neural network algorithms is a relatively new phenomenon, triggered by seminal works such as ImageNet [KSH12]. In computer graphics, such approaches have led to impressive results, e.g., for synthesizing novel viewpoints of natural scenes [FNPS16], to generate photorealistic face textures [SWH * 16], and to robustly transfer image styles between photographs [LPSB17], to name just a few examples. The underlying optimization approximates an unknown function f * (x) = y, by minimizing an associated loss function L such that f (x, θ) ≈ y.…”

Deep Painterly Harmonization

“…Gatys et al [GEB*17] add the possibility for users to guide the transfer with annotations. In the context of photographic transfer, Luan et al [LPSB17] limit mismatches using scene analysis. Li and Wand [LW16a] use nearest‐neighbor correspondences between neural responses to make the transfer content‐aware.…”

Lighting transfer across multiple views through local color transforms

“…In contrast, our result has lighting similar to that of the target image, and people and objects in shadow still have their colors. We show more comparison results with Shih et al [16] and deep photo style transfer [6] in Fig. S4 and Fig.…”

“…Li et al [5] recolor images using geodesic distance based on harmonization. More recently, Luan et al [6] propose a deep learning approach for photographic style transfer. These methods produce visually pleasing recolored images but cannot change local lighting.…”