Style Augmentation: Data Augmentation via Style Randomization

Abstract

We introduce style augmentation, a new form of data augmentation based on random style transfer, for improving the robustness of Convolutional Neural Networks (CNN) over both classification and regression based tasks. During training, style augmentation randomizes texture, contrast and color, while preserving shape and semantic content. This is accomplished by adapting an arbitrary style transfer network to perform style randomization, by sampling target style embeddings from a multivariate normal distribution instead of computing them from a style image. In addition to standard classification experiments, we investigate the effect of style augmentation (and data augmentation generally) on domain transfer tasks. We find that data augmentation significantly improves robustness to domain shift, and can be used as a simple, domain agnostic alternative to domain adaptation. Comparing style augmentation against a mix of seven traditional augmentation techniques, we find that it can be readily combined with them to improve network performance. We validate the efficacy of our technique with domain transfer experiments in classification and monocular depth estimation illustrating superior performance over benchmark tasks.

How to augment your ViTs? Consistency loss and StyleAug, a random style transfer augmentation

“…StyleAug differs from the work of Jackson et al (2019) in several ways. First, instead of performing style transfer to a different dataset (e.g., art or painting datasets), we performed style transfer to randomly chosen images within the same mini-batch (i.e., the same dataset).…”

“…Xu et al (2021) used a random convolution augmentation (to distort textures) combined with a consistency loss to improve CNN generalization to unseen domains such as ImageNet-sketch. Jackson et al (2019) used style transfer to paintings (Painter by Numbers) as an augmentation technique. When combined with traditional augmentations, their augmentation improved CNN classification and domain transfer performance on several small datasets, and on monocular depth estimation in the KITTI dataset.…”

The Role of the Input in Natural Language Video Description

“…In the recent work in [50], it is presented style augmentation as a novel strategy to perform visual data augmentation exploiting a style transfer network [51]. In particular, the texture, contrast, colour and illumination of the image is altered, but shapes and semantic content are preserved.…”

“…Content and style are modelled by two separate terms of the loss function, minimized to synthesise the new image having the desired style and content. Following the novel approach of [50], in this work style augmentation is tested for NLVD.…”

“…In addition, the videos have been transformed in the style of some artistic paintings. This was motivated by the fact that the strategy to perform data augmentation via style transfer has been found beneficial for many Computer Vision tasks [50]. In this study, the effectiveness of style augmentation for NLVD is investigated.…”