Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings

Abstract

Due to the digitization of fine art collections, pictures of fine art objects stored at museums and art galleries became widely available to the public. It created a demand for efficient software tools that would allow rapid retrieval and semantic categorization of art. This paper introduces a new, two-stage image classification approach aiming to improve the style classification accuracy. At the first stage, the proposed approach divides the input image into five patches and applies a deep convolutional neural network (CNN) to train and classify each patch individually. At the second stage, the outcomes from the individual five patches are fused in the decision-making module, which applies a shallow neural network trained on the probability vectors given by the first-stage classifier. While the first stage categorizes the input image based on the individual patches, the second stage infers the final decision label categorizing the artistic style of the analyzed input image. The key factor in improving the accuracy compared to the baseline techniques is the fact that the second stage is trained independently on the first stage using probability vectors instead of images. This way, the second stage is effectively trained to compensate for the potential mistakes made during the first stage. The proposed method was tested using six different pre-trained CNNs (AlexNet, VGG-16, VGG-19, GoogLeNet, ResNet-50, and Inceptionv3) as the first-stage classifiers, and a shallow neural network as a second-stage classifier. The experiments conducted using three standard art classification datasets indicated that the proposed method presents a significant improvement over the existing baseline techniques. INDEX TERMS Fine art style recognition, painting classification, machine learning, multi-stage classification, transfer learning, digital humanities.

Visual link retrieval and knowledge discovery in painting datasets

“…These characteristics can also be used to reveal the artist of a precise area of an artwork, in the case of multiple authorship of the same work. Encouraging results from the application of deep CNNs to artistic style classification have been recently reported in [11,24,30]. Other works, such as [35], have also experimented with CNN models trained with additional data, particularly time period, reporting better results.…”

An Information Theory Approach to Aesthetic Assessment of Visual Patterns

“…Due to the small number of images in the two datasets, the proposed approach cannot be compared to deep-learning-based approaches [ 47 , 48 , 49 , 50 , 51 ]. The proposed approach will be compared with three related approaches; the first one is based on the Birkhoff model [ 52 , 53 ], where Shannon entropy and image compressibility are used to represent the order and complexity of the Birkhoff model.…”

Poster-Based Multiple Movie Genre Classification Using Inter-Channel Features

“…Using this technique, the style losses between the target (original) and generated images during the style transfer were calculated, such that the styles and textures were similar between the target and generated images. The Gram matrix was extracted as a feature for images-such as famous paintings with particular styles-and used for classification [26], [27]. The Gram matrix can also be used for medical images as well as images with specific styles [28].…”