Object recognition in art drawings: Transfer of a neural network

Abstract

We consider the problem of recognizing objects in collections of art works, in view of automatically labeling, searching and organizing databases of art works. To avoid manually labelling objects, we introduce a framework for transferring a convolutional neural network (CNN), trained on available large collections of labelled natural images, to the context of drawings. We retrain both the top and the bottom layer of the network, responsible for the high-level classification output and the low-level features detection respectively, by transforming natural images into drawings. We apply this procedure to the drawings in the Jan Brueghel Wiki, and show the transferred CNN learns a discriminative metric on drawings and achieves good recognition accuracy. We also discuss why standard descriptorbased methods is problematic in the context of drawings.

Weakly Supervised Object Detection in Artworks

“…In [11], it is shown that recycling CNNs directly for the task of recognising objects in paintings, without fine-tuning, yields surprisingly good results. Similar conclusions were also given in [55] for artistic drawings. In [33], a robust low rank parametrized CNN model is proposed to recognise common categories in an unseen domain (photo, painting, cartoon or sketch).…”

“…Several recent works show that recycling analysis tools that have been developed for natural images (photographs) can yield surprisingly good results for analysing paintings or drawings. In particular, impressive classification results are obtained on painting databases by using convolutional neural networks (CNNs) designed for the classification of photographs [11,55]. These results occur in a general context were methods of transfer learning [15] (changing the task a model was trained for) and domain adaptation (changing the nature of the data a model was trained on) are increasingly applied.…”

Discovering Visual Patterns in Art Collections With Spatially-Consistent Feature Learning

“…Others focus on applying and generalizing visual correspondence and object detection methods to paintings using both classical [41,18,7,4], as well as deep [6,8,47,20] methods. Most closely related to us is work of Yin et al [49], which used the same Brueghel data [1], annotating it to train detectors for five object categories (carts, cows, windmills, rowboats and sailbaots).…”

An analysis of the transfer learning of convolutional neural networks for artistic images

“…This approach is by far the most used one. For the domain of artistic images, it has been used for style classification (2,12,27), object recognition in drawings (30) or iconographic characters (13), people detection across a variety of artworks (28), visual link retrieval (21), author classification (15,20) or several of those tasks at the same time (1).…”

“…7). Note that in (30), it has been shown that the low-level layer filters have been modified by a fine-tuning on an almost monochrome drawing training dataset. This suggests that the statistics of painting images are closer to those of natural images than those of drawing ones.…”