Attention-Aware Multi-Stroke Style Transfer

Abstract

Neural style transfer has drawn considerable attention from both academic and industrial field. Although visual effect and efficiency have been significantly improved, existing methods are unable to coordinate spatial distribution of visual attention between the content image and stylized image, or render diverse level of detail via different brush strokes. In this paper, we tackle these limitations by developing an attention-aware multi-stroke style transfer model. We first propose to assemble self-attention mechanism into a style-agnostic reconstruction autoencoder framework, from which the attention map of a content image can be derived. By performing multi-scale style swap on content features and style features, we produce multiple feature maps reflecting different stroke patterns. A flexible fusion strategy is further presented to incorporate the salient characteristics from the attention map, which allows integrating multiple stroke patterns into different spatial regions of the output image harmoniously. We demonstrate the effectiveness of our method, as well as generate comparable stylized images with multiple stroke patterns against the state-of-theart methods.

Arbitrary Style Transfer via Multi-Adaptation Network

“…To expand the application of style transfer further, many works focus on arbitrary style transfer methods [6,9,16,17,20,23,25,28]. AdaIN [8] and WCT [17] align the second-order statistics of style image to content image.…”

“…However, the holistic style transfer process makes the generated quality disappointing. Patch-swap based methods [1,28] aim to transfer style image patches to content image according to the similarity between patches pairs. However, when the distributions of content and style structure vary greatly, few style patterns are transferred to the content image through style-swap [1].…”

“…However, when the distributions of content and style structure vary greatly, few style patterns are transferred to the content image through style-swap [1]. Yao et al [28] improved the style-swap [1] method by adding multi-stroke control and self-attention mechanism. Inspired by self-attention mechanism, Park et al [20] proposed styleattention to match the style features onto the content features, but may encounter semantic structures of content image distorting problem.…”

Anisotropic Stroke Control for Multiple Artists Style Transfer

“…Style transfer results. We evaluate our approach with five stateof-the-art methods: AdaIN [14], Style-Aware [32], AAMS [40], Style Swap [7] and WCT [22]. Noting that style aware includes two subsequent works [20,21] and their effects are not much different.…”

“…Inspired by human perception, recent methods such as [25,40,42] incorporate attention mechanism to achieve the different generation granularity in non-trivial regions and trivial regions. Yao et al [40] follow this scheme and introduce self-attention module which generates a salient map in hidden space, and then adjusts the style-stroke of different regions according to that map. However, this method suffers from the uncertainty of focus areas and poor effect of highlighting the pixel-wise salient part of semantic content.…”