Learning Linear Transformations for Fast Image and Video Style Transfer

Abstract

Given a random pair of images, an arbitrary style transfer method extracts the feel from the reference image to synthesize an output based on the look of the other content image. Recent arbitrary style transfer methods transfer second order statistics from reference image onto content image via a multiplication between content image features and a transformation matrix, which is computed from features with a pre-determined algorithm. These algorithms either require computationally expensive operations, or fail to model the feature covariance and produce artifacts in synthesized images. Generalized from these methods, in this work, we derive the form of transformation matrix theoretically and present an arbitrary style transfer approach that learns the transformation matrix with a feed-forward network. Our algorithm is highly efficient yet allows a flexible combination of multi-level styles while preserving content affinity during style transfer process. We demonstrate the effectiveness of our approach on four tasks: artistic style transfer, video and photo-realistic style transfer as well as domain adaptation, including comparisons with the stateof-the-art methods.

A Closed-Form Solution to Universal Style Transfer

“…2. This has also been pointed out by [18]. Theoretically, there are infinite solutions, considering only Eq.…”

“…[12] uses AdaIN as the feature transform and trains the decoder over large collections of content and style images. [18] trains two meta networks for the whitening and coloring matrices, following the formulation of WCT [19]. Many other works also extend neural style transfer to video [2,11,27] and stereoscopic style transfer [4].…”

“…WCT [19] proposes to use feature whitening and coloring to match the covariance matrices of style and content images. However, as pointed out by [18], WCT is not the only approach to matching the covariance matrices. [29] proposes a method to combine patch match with WCT and AdaIN.…”

“…It does not consider the content loss and cannot well preserve the image structure. Moreover, multiplying an orthogonal matrix between the whitening and coloring matrices can also match the covariance matrices, which has been pointed out by [18]. [20] shows that matching Gram matrices is equivalent to minimizing the Maximum Mean Discrepancy (MMD) with the second order polynomial kernel.…”

“…Assuming the transformation is linear, we find that both AdaIN and WCT are special cases of our formulation. Although [18] also assumes the transformation is linear, it still follows the whitening and coloring pipeline and trains two meta networks for whitening and coloring matrices. Contrary to that, we directly find the transformation under the optimal transport formulation.…”