Lightweight Photo-Realistic Style Transfer for Mobile Devices

Abstract

Neural Style Transfer enables synthesizing an image in the style of a reference image with convolutional neural networks. Photo-realistic style transfer aims to retain the photorealism in the synthesized image. Many of the state-of-the-art photo-realistic style transfer techniques uses networks like VGG as the backbone effectively making it computationally expensive and infeasible for deployment in systems with limited resources like mobile devices. In this work, it is shown that lightweight VGG like networks can be trained with knowledge distillation technique to achieve similar performance with *35x lesser model parameters. In addition, a novel and efficient method is proposed to train neural networks to learn the image smoothing operator to enhance the photorealism in the stylized images. The proposed improvements, thus makes it feasible to achieve effective and fast on-device photo-realistic style transfer.