Personalised Aesthetics with Residual Adapters

Abstract

The use of computational methods to evaluate aesthetics in photography has gained interest in recent years due to the popularization of convolutional neural networks and the availability of new annotated datasets. Most studies in this area have focused on designing models that do not take into account individual preferences for the prediction of the aesthetic value of pictures. We propose a model based on residual learning that is capable of learning subjective, userspecific preferences over aesthetics in photography, while surpassing the stateof-the-art methods and keeping a limited number of user-specific parameters in the model. Our model can also be used for picture enhancement, and it is suitable for content-based or hybrid recommender systems in which the amount of computational resources is limited.The problem of taking into account subjective preferences on image aesthetics prediction is referred to as personalized image aesthetics [27]. Most recent approaches to image aesthetics evaluation have used different deep-learning models, which require a significant amount of annotated data for their training and evaluation. In real-world situations, it is unrealistic to assume that we will have thousands of annotated examples of rated images for any given user. This puts limits on the use of deep learning models for personalized image aesthetics prediction.In order to train a machine learning model capable of taking into account individual preferences over aesthetics in photography, an annotated dataset with the identities of the raters of each picture is needed. One example of this kind of dataset is the FLICKER-AES dataset, presented by Ren et al. [27], which contains over 40000 images rated by more than 200 different human raters. Their study provides, along with this dataset (and another, smaller, dataset), a residual-based learning model capable of taking into account user-specific preferences over aesthetics in photography.We build on their work, and propose an end-to-end convolutional neural network model capable of modelling user-specific preferences with different levels of abstraction, while keeping a reduced number of user-specific parameters within the model. Our method models user-specific preferences by using residual adapters, which were presented in [26,25] and have shown success in multi-domain learning. The main difference between our model and Ren et al.'s is that they model user-specific preferences by first training a generic aesthetics network, which predicts a mean aesthetic score, and computes a user-specific offset by training a Support Vector Regressor using the predicted content and some manually-defined attributes of the picture as its input; whereas our model embeds the user-specific parameters in different layers of the neural network, therefore allowing the model to find user-specific features with different levels of abstraction, and which do not necessarily depend on the contents and a fixed set of attributes of the pictures.Our main contributions are as follows: First, we propo...

SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

“…To achieve this, and, in order to account for artifacts in the borders of the generated textures, we tile the synthesized images until they cover the same resolution as their corresponding inputs. Interestingly, methods based on patches [13], [22], [45] obtain better SSIM and Si-FID scores than previous deep learning-based methods [16], [23]. This difference is not seen in the LPIPS metric.…”

“…While traditional methods used handcrafted features [39], [40], recent parametric methods rely on deep neural networks as their parameterization. Activations within latent spaces in pre-trained CNNs have shown to capture relevant statistics of the style and texture of images [41], [42], [43], [44], [45]. Textures can be synthesized through this approach by gradientdescent optimization [46], [47] or by training a neural network that learns those features [23], [48].…”

How Will It Drape Like? Capturing Fabric Mechanics from Depth Images

“…Pre‐training an image descriptor model on contrastive or self‐supervised learning tasks, and use the activations of its last layer as input to the downstream task ( Linear Probing ) [CKNH20, RKH * 21, CXH21, HCX * 22, KRJ * 22]. For domain adaptation problems, it is common to adapt the internal representations of pre‐trained CNNs so as to efficiency [RBV17, RBV18, RPB19, PCYS20, LLB22]. Inspired by these approaches, we design a model that leverages fine‐tuning of a pre‐trained image CNN classifier as a feature extractor, capable of processing depth images, and extend it to account for additional input variables, and handling multiple images at the same time during test.…”

A Deeper Look at Human Visual Perception of Images

“…However, despite this expected variability when evaluating subjective properties of images, we show a sufficiently large degree of consistency between different users' judgments, suggesting it is possible to devise automatic systems to estimate visual sentiment directly from images, ignoring user differences. One of our future works is to design personalized approaches to predict both human sentiment and individual differences [73,74].…”