Leveraging Expert Feature Knowledge for Predicting Image Aesthetics

Abstract

The ability to rank images based on their appearance finds many real-world applications such as image retrieval or image album creation. Despite the recent dominance of deep learning methods in computer vision which often result in superior performance, they are not always the methods of choice because they lack interpretability. In this work, we investigate the possibility of improving image aesthetic inference of convolutional neural networks with hand-designed features that rely on domain expertise in various fields. We perform a comparison of hand-crafted feature sets in their ability to predict fine-grained aesthetics scores on two image aesthetics datasets. We observe that even feature sets published earlier are able to compete with more recently published algorithms and, by combining the algorithms together, one can obtain a significant improvement in predicting image aesthetics. By using a tree-based learner, we perform feature elimination to understand the best performing features overall and across different image categories. Only roughly 15 % and 8 % of the features are needed to achieve full performance in predicting a fine-grained aesthetic score and binary classification respectively. By combining hand-crafted features with meta-features that predict the quality of an image based on CNN features, the model performs better than a baseline VGG16 model. One can, however, achieve more significant improvement in both aesthetics score prediction and binary classification by fusing the hand-crafted features and the penultimate layer activations. Our experiments indicate an improvement up to 2.2 % achieving current state-of-the-art binary classification accuracy on the AVA dataset when the hand-designed features are fused with activation from VGG16 and ResNet50 networks.

And the nominees are: Using design-awards datasets to build computational aesthetic evaluation model

“…In the existing aesthetic computing research, there are three main types of studies, the aesthetic ranking analysis [13], classification of aesthetic level (low/high or positive/negative) [15,18,24,[26][27][28][29] and the aesthetic score prediction [17,31,43]. In the majority of the related works, researchers conducted classification method on image aesthetic computing study.…”

“…Works that attempt to bridge computing and the perception of aesthetics have three main tasks. First, aesthetic images are to be evaluated using qualitative measures [16][17][18], including user interface design [19][20][21], photos, paintings, and filmed scenes. Second, multimedia retrieval is to be developed, based on aesthetic recognition.…”

“…The largest aesthetic database is AVA database (Aesthetic Visual Analysis) [42], which is a widely used aesthetic database containing 250,000 images in 60 classes. The other aesthetic databases include FLICKR-AES [17], HB [18], CUHK [29], PNE [29], MSR-ICD [37] and CUHKPQ [39].…”

“…VGG-19. VGG-19 can achieve great accuracy in the large-scale image recognition [48] and it is also applicable in a number of image aesthetic computing studies [18][39] [43]. It is using architecture with small convolution filters.…”

“…ResNet-50. ResNet-50 is applied to extract multimodal features of design work images in view of its feasibility in existing studies [18,39]. The ResNet-50 training process is shown as follows:…”