Benchmarking non-photorealistic rendering of portraits

Abstract

Despite the recent upsurge of activity in image-based non-photorealistic rendering (NPR), and in particular portrait image stylisation, due to the advent of neural style transfer, the state of performance evaluation in this field is limited, especially compared to the norms in the computer vision and machine learning communities. Unfortunately, the task of evaluating image stylisation is thus far not well defined, since it involves subjective, perceptual and aesthetic aspects. To make progress towards a solution, this paper proposes a new structured, three level, benchmark dataset for the evaluation of stylised portrait images. Rigorous criteria were used for its construction, and its consistency was validated by user studies. Moreover, a new methodology has been developed for evaluating portrait stylisation algorithms, which makes use of the different benchmark levels as well as annotations provided by user studies regarding the characteristics of the faces. We perform evaluation for a wide variety of image stylisation methods (both portrait-specific and general purpose, and also both traditional NPR approaches and neural style transfer) using the new benchmark dataset.

Watercolour Rendering of Portraits

“…Berger et al [3] learned models of stroke parameters from training examples which consisted of line sketches at four abstraction levels that were provided by seven artists. The method by Rosin and Lai [21] first stylises the image with abstracted regions of flat colours plus black and white lines [15]. Facial features are localised in the input image.…”

“…Recent work [11] has applied style transfer to videos of faces, and uses a semantic segmentation guide to control the synthesis. In an attempt to analyse the performance of NPR portrait algorithms, Rosin et al [22] recently introduced a benchmark data set of 40 images, split into two levels. The first level is constrained to contain frontal views of adult faces with neutral expressions, and uncluttered background.…”

“…Recently several state-of-the-art portrait-specific NPR algorithms were applied to a benchmark set of portraits [22], and it was notable that all of them ensured that the eyes were the most carefully rendered parts of the image. In addition to performing rendering at a higher resolution for the eyes, other elements in the processing can ensure accuracy.…”

“…However, the size threshold for cleaning was reduced for the eye regions to avoid the possibility of removing critical detail. Another example is given in [21] where an artificial highlight was added to the eye in case the real highlight was absent due to extremely low contrast in the input image. In the more general area of photo enhancement special attention is also paid to eyes [23].…”

“…In addition, techniques for stylistic elements of watercolour such as wobbling, glazing and diffusion are introduced. Results are shown on images from the NPRportrait benchmark [22].…”