In Search of Art

Abstract

The objective of this work is to find objects in paintings by learning object-category classifiers from available sources of natural images. Finding such objects is of much benefit to the art history community as well as being a challenging problem in large-scale retrieval and domain adaptation.We make the following contributions: (i) we show that object classifiers, learnt using Convolutional Neural Networks (CNNs) features computed from various natural image sources, can retrieve paintings containing these objects with great success; (ii) we develop a system that can learn object classifiers on-the-fly from Google images and use these to find a large variety of previously unfound objects in a dataset of 210,000 paintings; (iii) we combine object classifiers and detectors to align objects to allow for direct comparison; for example to illustrate how they have varied over time.

The Art of Detection

“…This include vehicles (boats, cars), animals (elephants, dogs), clothes (uniform, gown), structures (cottage, church), parts of structures (spires, roof) among others. Performance is evaluated quantitatively as a classification-by-detection problem as in section 5: we rank each of the 210,000 paintings according to the score corresponding to its highest scoring object region and by eye, compute Pre@k -Precision at k, the fraction of the top-k retrieved paintings that contain This system is crucially able to overcome one of the difficulties experienced by our image-level classification system [12]: a notable difference in performance occurs when an object is large in natural images and small in paintings. A good examples of this is 'wheel'.…”

“…However, these objects are limited to those of PASCAL VOC, which isn't very useful if an art historian is interested in search for depictions of fruit or elephants. To accommodate for this, we provide a live system, inspired by [7,9,12] where a user may supply a query, and paintings are retrieved that contain the object with its bounding box provided. This improves on our imagelevel painting retrieval system [12] in two ways: Firstly, it retrieves small objects that cannot be located at image-level.…”

“…With increasingly powerful image representations provided by each generation of Convolutional Neural Networks (CNNs) there has been a steady increase in performance over a variety of challenging datasets of natural, photographic images [17,31,34] (and for a variety of tasks [15,22,29,32]). It has been shown that these representations transfer well between domains such as between DSLR and webcam images [39], natural images and sketches [43] and of particular interest to us, between images and paintings [12,11].…”

Visual Link Retrieval in a Database of Paintings

“…In [12], the authors show that modern object classification frameworks based on convolutional neural network perform relatively well on paintings data. That way, they can have the user search for an object category in large collection of paintings from a simple text query.…”

“…The fact that the migration of patterns in paintings is mainly important in the Modern Period (1400-1800) is an important factor in choosing our base [12,13] for object classification. -RKD Challenge [25]: coming from the Rijksmuseum, this benchmark was created for scientists to test their algorithms on artists identification, labelling of materials and estimating the creation year.…”