A Closed-Form Solution to Universal Style Transfer

Abstract

Universal style transfer tries to explicitly minimize the losses in feature space, thus it does not require training on any pre-defined styles. It usually uses different layers of VGG network as the encoders and trains several decoders to invert the features into images. Therefore, the effect of style transfer is achieved by feature transform. Although plenty of methods have been proposed, a theoretical analysis of feature transform is still missing. In this paper, we first propose a novel interpretation by treating it as the optimal transport problem. Then, we demonstrate the relations of our formulation with former works like Adaptive Instance Normalization (AdaIN) and Whitening and Coloring Transform (WCT). Finally, we derive a closed-form solution named Optimal Style Transfer (OST) under our formulation by additionally considering the content loss of Gatys. Comparatively, our solution can preserve better structure and achieve visually pleasing results. It is simple yet effective and we demonstrate its advantages both quantitatively and qualitatively. Besides, we hope our theoretical analysis can inspire future works in neural style transfer. Code is available at https://github.com/lu-m13/ OptimalStyleTransfer.

Preserving Global and Local Temporal Consistency for Arbitrary Video Style Transfer

“…• Furthermore, the texture in our stylized frames is more authentic, which validates that our method performs better than ASTV in maintaining the detailed content of the source video. Figure 4 shows two examples of two consecutive frames from CutBunny video that are stylized with Mondrian style image by the image style transfer methods (i.e., AdaIN [9] ,OST [13] and the base model [12]) and our method. The first two columns demonstrate the object motion and the last two columns show the static scene.…”

“…The MPI Sintel dataset provides multiple real-world scenarios, which contains 35 videos. The WikiArt dataset [15] is used as the style image dataset, consisting of 11,025 images, and all the test style images are from published implementations [13].…”

“…Compared methods. We compare our method with several existing video style transfer methods such as [3,5,8,19] and image style transfer methods such as [9,12,13]. We use the model proposed in [12] as our based model.…”

“…• AdaIN[9] is a classical arbitrary image style transfer method that adjusts the mean and variance of the content input to match those of the style input. • OST[13] focuses on the theoretical analysis of feature transform and proposes a new closed-form solution.•[12] learns a transformation matrix that is efficient for arbitrary image style transfer.…”

Semantic Context-Aware Image Style Transfer

“…The transformed features are then fed forward into decoder layers to obtain the stylized image. To further improve [12] where only the style loss is considered, Lu et al [13] seek optimal style transfer to preserve image structures by considering the content loss. Li et al [14] propose a learnable linear transformation matrix based on arbitrary pairs of content and style images by two light-weighted CNNs.…”