Learning to Warp for Style Transfer

Abstract

Since its inception in 2015, Style Transfer has focused on texturing a content image using an art exemplar. Recently, the geometric changes that artists make have been acknowledged as an important component of style [42,55,62,63]. Our contribution is to propose a neural network that, uniquely, learns a mapping from a 4D array of inter-feature distances to a non-parametric 2D warp field. The system is generic in not being limited by semantic class, a single learned model will suffice; all examples in this paper are output from one model.Our approach combines the benefits of the high speed of Liu et al. [42] with the non-parametric warping of Kim et al. [55]. Furthermore, our system extends the normal NST paradigm: although it can be used with a single exemplar, we also allow two style exemplars: one for texture and another for geometry. This supports far greater flexibility in use cases than single exemplars can provide.

Industrial Style Transfer with Large-scale Geometric Warping and Content Preservation

“…Later, the convolutional neural network has been popular in geometric matching due to its ability to extract powerful and robust features. The current best methods follow the network paradigm proposed by [46] which consists of feature extraction, matching layer and regression network, and make various improvements [18,27,38,46,47] based on it. All the above methods act on two RGB images and attempt to estimate a warp field to directly match them.…”

“…However, it is difficult to quickly create high-quality product appearances due to the human intelligence in the VPD process, which is heavily dependent on designers' creative ability. Fortunately, neural style transfer (NST) [16,21,28,38], aiming at transferring artistic and geometric styles of one or two reference images to a content image, has a strong opportunity to assist the designers because the art style transformation is suitable for the aesthetic value, and some geometric shape transformations can gain the functional and symbolic values, such as Beijing National Stadium (bird's nest and building). Therefore, we seek for a style transfer formulation to automatically generate many visual appearance candidates of new products for industrial designers' reference.…”

“…However, most modern NST methods [14,25,32,59,60], including geometric NST [28,38], are difficultly or impossibly extended into directly design visual product appearances due to the following two challenges. One is the large- scale geometric shape between diverse objects (or products) since designing new products is often to fuse two objects with very different geometries, such as flying cars (airplanes and cars) and butterfly doors (butterfly's wings and car doors).…”

“…In contrast to the existing NST methods, InST consists of large-scale geometric warping (LGW) and interestconsistency texture transfer (ICTT). Unlike small-scale geometric NST [28,38], LGW designs a neural warping field between the shape masks of the source and target products using a shape-consistency loss. This is different from a warping field between their textural pixels as it results in a worse optimization, that is, a failure deformation.…”

CAMS: Color-Aware Multi-Style Transfer

“…Inspired by Deep Dream, Gatys et al, [3] proposed a neural style transfer (NST) method to minimize statistical differences of deep features, extracted from intermediate layers of a pre-trained CNN (e.g., VGG net [21]), of content and style images. After the impressive results achieved by the NST work in [3], many methods have been proposed to perform style transfer leveraging the power of CNNs (e.g., [4,5,11,13,14,16,18,19,22,23,24,25,26,27]).…”