Classification of Style in Fine-Art Paintings Using Transfer Learning and Weighted Image Patches

Sandoval, Catherine; Lech, Margaret; Pirogova, Elena

doi:10.1109/icspcs.2018.8631731

Cited by 18 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in our previous work [27], a significant improvement of the classification results can be achieved using a weighted sum of outcomes for individual patches. We have introduced an optimized approach, where individual patches were independently classified by a CNN, and then a weighted average of the classification outcomes for individual patches was estimated to determine the final style label.…”

Section: Related Workmentioning

confidence: 83%

See 1 more Smart Citation

Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings

2019

Self Cite

View full text Add to dashboard Cite

Due to the digitization of fine art collections, pictures of fine art objects stored at museums and art galleries became widely available to the public. It created a demand for efficient software tools that would allow rapid retrieval and semantic categorization of art. This paper introduces a new, two-stage image classification approach aiming to improve the style classification accuracy. At the first stage, the proposed approach divides the input image into five patches and applies a deep convolutional neural network (CNN) to train and classify each patch individually. At the second stage, the outcomes from the individual five patches are fused in the decision-making module, which applies a shallow neural network trained on the probability vectors given by the first-stage classifier. While the first stage categorizes the input image based on the individual patches, the second stage infers the final decision label categorizing the artistic style of the analyzed input image. The key factor in improving the accuracy compared to the baseline techniques is the fact that the second stage is trained independently on the first stage using probability vectors instead of images. This way, the second stage is effectively trained to compensate for the potential mistakes made during the first stage. The proposed method was tested using six different pre-trained CNNs (AlexNet, VGG-16, VGG-19, GoogLeNet, ResNet-50, and Inceptionv3) as the first-stage classifiers, and a shallow neural network as a second-stage classifier. The experiments conducted using three standard art classification datasets indicated that the proposed method presents a significant improvement over the existing baseline techniques. INDEX TERMS Fine art style recognition, painting classification, machine learning, multi-stage classification, transfer learning, digital humanities.

show abstract

Section: Related Workmentioning

confidence: 83%

“…The optimal weight values were derived by a numerical optimization algorithm that aimed to maximize the overall system classification accuracy. Details of this approach can be found in [27].…”

Section: ) Scenario 5 -Weighted Averagementioning

confidence: 99%

Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…For this analysis, we compare model performance from different layers with human performance for different fragment sizes. We emphasize that values for the model are not obtained by presenting the model with fragments (as for example in Rodriguez et al (2018) ): here the model is always presented with full-size images. Different values refer to different classifiers at different depths.…”

Section: Resultsmentioning

confidence: 99%

“…The exact mechanisms underlying orientation judgements are not fully understood. Some authors have suggested that the perception of orientation depends more on low-level stimulus properties than higher level object recognition and/or image interpretation ( Lindauer, 1987 ), prompting others to investigate the potential role of relatively simple cues, such as Fourier amplitude spectrum slope ( Mather, 2012 ), or image statistics based on explicit rules gathered from several art theories incorporated into a machine learning algorithm ( Liu et al., 2017 ) (see Elgammal et al., 2018 ; Rodriguez et al., 2018 for related applications).…”

Section: Introductionmentioning

confidence: 99%

A deep-learning framework for human perception of abstract art composition

Lelièvre

Neri

2021

Journal of Vision

View full text Add to dashboard Cite

Artistic composition (the structural organization of pictorial elements) is often characterized by some basic rules and heuristics, but art history does not offer quantitative tools for segmenting individual elements, measuring their interactions and related operations. To discover whether a metric description of this kind is even possible, we exploit a deep-learning algorithm that attempts to capture the perceptual mechanism underlying composition in humans. We rely on a robust behavioral marker with known relevance to higher-level vision: orientation judgements, that is, telling whether a painting is hung "right-side up." Humans can perform this task, even for abstract paintings. To account for this finding, existing models rely on "meaningful" content or specific image statistics, often in accordance with explicit rules from art theory. Our approach does not commit to any such assumptions/schemes, yet it outperforms previous models and for a larger database, encompassing a wide range of painting styles. Moreover, our model correctly reproduces human performance across several measurements from a new web-based experiment designed to test whole paintings, as well as painting fragments matched to the receptive-field size of different depths in the model. By exploiting this approach, we show that our deep learning model captures relevant characteristics of human orientation perception across styles and granularities. Interestingly, the more abstract the painting, the more our model relies on extended spatial integration of cues, a property supported by deeper layers.

show abstract

“…Common problems are confusing backgrounds, the density of instances, less discriminative class features, large intra-class diversity and inter-class similarity, artificial colours and shadows, and low image quality. Finally, most state-of-the-art end-to-end WSOD methods freeze a large part of the network, thus making it impossible to take advantage of the renowned benefits of Transfer Learning (TL) on non-natural data sets [ 13 , 14 , 15 ].…”

Section: Introductionmentioning

confidence: 99%

Proposals Generation for Weakly Supervised Object Detection in Artwork Images

2022

View full text Add to dashboard Cite

Object Detection requires many precise annotations, which are available for natural images but not for many non-natural data sets such as artworks data sets. A solution is using Weakly Supervised Object Detection (WSOD) techniques that learn accurate object localization from image-level labels. Studies have demonstrated that state-of-the-art end-to-end architectures may not be suitable for domains in which images or classes sensibly differ from those used to pre-train networks. This paper presents a novel two-stage Weakly Supervised Object Detection approach for obtaining accurate bounding boxes on non-natural data sets. The proposed method exploits existing classification knowledge to generate pseudo-ground truth bounding boxes from Class Activation Maps (CAMs). The automatically generated annotations are used to train a robust Faster R-CNN object detector. Quantitative and qualitative analysis shows that bounding boxes generated from CAMs can compensate for the lack of manually annotated ground truth (GT) and that an object detector, trained with such pseudo-GT, surpasses end-to-end WSOD state-of-the-art methods on ArtDL 2.0 (≈41.5% mAP) and IconArt (≈17% mAP), two artworks data sets. The proposed solution is a step towards the computer-aided study of non-natural images and opens the way to more advanced tasks, e.g., automatic artwork image captioning for digital archive applications.

show abstract

Classification of Style in Fine-Art Paintings Using Transfer Learning and Weighted Image Patches

Cited by 18 publications

References 18 publications

Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings

Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings

A deep-learning framework for human perception of abstract art composition

Proposals Generation for Weakly Supervised Object Detection in Artwork Images

Contact Info

Product

Resources

About