Fine-tuning Convolutional Neural Networks for fine art classification

Abstract

The increasing availability of large digitized fine art collections opens new research perspectives in the intersection of artificial intelligence and art history. Motivated by the successful performance of Convolutional Neural Networks (CNN) for a wide variety of computer vision tasks, in this paper we explore their applicability for art-related image classification tasks. We perform extensive CNN fine-tuning experiments and consolidate in one place the results for five different art-related classification tasks on three large fine art datasets. Along with addressing the previously explored tasks of artist, genre, style and time period classification, we introduce a novel task of classifying artworks based on their association with a specific national artistic context. We present state-of-the-art classification results of the addressed tasks, signifying the impact of our method on computational analysis of art, as well as other image classification related research areas. Furthermore, in order to question transferability of deep representations across various source and target domains, we systematically compare the effects of domain-specific weight initialization by evaluating networks pre-trained for different tasks, varying from object and scene recognition to sentiment and memorability labelling. We show that fine-tuning networks pre-trained for scene recognition and sentiment prediction yields better results than fine-tuning networks pre-trained for object recognition. This novel outcome of our work suggests that the semantic correlation between different domains could be inherent in the CNN weights. Additionally, we address the practical applicability of our results by analysing different aspects of image similarity. We show that features derived from fine-tuned networks can be employed to retrieve images similar in either style or content, which can be used to enhance capabilities of search systems in different online art collections.

Visual link retrieval and knowledge discovery in painting datasets

“…These characteristics can also be used to reveal the artist of a precise area of an artwork, in the case of multiple authorship of the same work. Encouraging results from the application of deep CNNs to artistic style classification have been recently reported in [11,24,30]. Other works, such as [35], have also experimented with CNN models trained with additional data, particularly time period, reporting better results.…”

Compare the performance of the models in art classification

“…The recent successes of deep CNNs in solving computer vision tasks hinges on the availability of large hand-labeled datasets such as ImageNet [5], which contains more than 15 million hand-labeled high-resolution images representing approximately 22,000 different object categories. In the art classification field, the authors of [6] used CaffeNet [7], which is a slightly modified version of the AlexNet model [8], to evaluate the fine-tuning process using five different pretrained networks. Some of these innovations focus on the way data are imported and the various models used.…”

Prediction of Tourist Flow Based on Deep Belief Network and Echo State Network

“…With the advent of artificial intelligence (AI) technologies like neural networks (NNs) [11,12] and fuzzy theory [13], data-driven methods have attracted more and more attention from scholars. The data-driven methods autonomously learn the historical data on tourist flow, especially the nonlinear, dynamical changes in the data.…”