Cross-depiction problem: Recognition and synthesis of photographs and artwork

Hall, Peter M.; Cai, Hongping; Wu, Qi; Corradi, Tadeo

doi:10.1007/s41095-015-0017-1

Cited by 39 publications

(29 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, such visual classifiers are generally trained on natural images, for which there is a copious amount of annotation (and which is often lacking for paintings). Unfortunately, as Hall et al observe [24], there is a drop in performance in training on natural images rather than paintings. So we ask, when it comes to classifying paintings using natural images as training data, what are we missing?…”

Section: Introductionmentioning

confidence: 99%

The Art of Detection

Crowley

Zisserman

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The objective of this work is to recognize object categories in paintings, such as cars, cows and cathedrals. We achieve this by training classifiers from natural images of the objects. We make the following contributions: (i) we measure the extent of the domain shift problem for image-level classifiers trained on natural images vs paintings, for a variety of CNN architectures; (ii) we demonstrate that classificationby-detection (i.e. learning classifiers for regions rather than the entire image) recognizes (and locates) a wide range of small objects in paintings that are not picked up by image-level classifiers, and combining these two methods improves performance; and (iii) we develop a system that learns a region-level classifier on-the-fly for an object category of a user's choosing, which is then applied to over 60 million object regions across 210,000 paintings to retrieve localised instances of that category.

show abstract

Section: Introductionmentioning

confidence: 99%

The Art of Detection

Crowley

Zisserman

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…[9] show that as paintings become more abstract, the performance of person detection degrades. [14,38] benchmark existing methods on artistic objects, but only propose improvements over older, non-convolutional techniques, which are greatly outperformed by neural networks. [37] provide a large dataset of artistic domains with per-image object annotations, but we found the labels were too coarse (due to multiple objects in the same image and no bounding box annotations) and the human annotations too sparse, to reliably perform recognition.…”

Section: Related Workmentioning

confidence: 99%

“…We downloaded 1,391 paintings 1 which cover eight of our ten categories (except "giraffe" and "zebra"), and annotated these images with 2,834 bounding boxes. Note that another painting dataset exists [14] but it only contains four of the ten classes we consider. To maintain the same set of domains as those used in PACS, we also include 12,008 sketches from the Sketchy Database [31].…”

Section: Datasetsmentioning

confidence: 99%

Artistic Object Recognition by Unsupervised Style Adaptation

Thomas

Kovashka

2019

Computer Vision – ACCV 2018

View full text Add to dashboard Cite

Computer vision systems currently lack the ability to reliably recognize artistically rendered objects, especially when such data is limited. In this paper, we propose a method for recognizing objects in artistic modalities (such as paintings, cartoons, or sketches), without requiring any labeled data from those modalities. Our method explicitly accounts for stylistic domain shifts between and within domains. To do so, we introduce a complementary training modality constructed to be similar in artistic style to the target domain, and enforce that the network learns features that are invariant between the two training modalities. We show how such artificial labeled source domains can be generated automatically through the use of style transfer techniques, using diverse target images to represent the style in the target domain. Unlike existing methods which require a large amount of unlabeled target data, our method can work with as few as ten unlabeled images. We evaluate it on a number of cross-domain object and scene classification tasks and on a new dataset we release. Our experiments show that our approach, though conceptually simple, significantly improves the accuracy that existing domain adaptation techniques obtain for artistic object recognition.

show abstract

“…Classifiers (deep CNNs) trained on real data are able to locate objects such as cars, cows and cathedrals (Crowley & Zisserman, 2016) if the artistic rendering is not very abstract. The problem of detecting/recognizing objects in any type of data, regardless if it is real or artistic, was named crossdepiction by Hall et al (Hall et al, 2015); however the problem is noted as being particular difficult and in the light of dedicated benchmarks , the results show plenty of space for improvement. A significant conclusion that arises is that all solutions that showed some degree of success did it for older artistic movements, where scene depiction was without particular abstraction.…”

Section: Related Workmentioning

confidence: 99%

Can we teach computers to understand art? Domain adaptation for enhancing deep networks capacity to de-abstract art

Badea

Florea

et al. 2018

Image and Vision Computing

View full text Add to dashboard Cite

Humans comprehend a natural scene at a single glance; painters and other visual artists, through their abstract representations, stressed this capacity to the limit. The performance of computer vision solutions matched that of humans in many problems of visual recognition. In this paper we address the problem of recognizing the genre (subject) in digitized paintings using Convolutional Neural Networks (CNN) as part of the more general dealing with abstract and/or artistic representation of scenes. Initially we establish the state of the art performance by training a CNN from scratch. In the next level of evaluation, we identify aspects that hinder the CNNs' recognition, such as artistic abstraction. Further, we test various domain adaptation methods that could enhance the subject recognition capabilities of the CNNs. The evaluation is performed on a database of 80,000 annotated digitized paintings, which is tentatively extended with artistic photographs, either original or stylized, in order to emulate artistic representations. Surprisingly, the most efficient domain adaptation is not the neural style transfer. Finally, the paper provides an experiment-based assessment of the abstraction level that CNNs are able to achieve.

show abstract

Cross-depiction problem: Recognition and synthesis of photographs and artwork

Cited by 39 publications

References 57 publications

The Art of Detection

The Art of Detection

Artistic Object Recognition by Unsupervised Style Adaptation

Can we teach computers to understand art? Domain adaptation for enhancing deep networks capacity to de-abstract art

Contact Info

Product

Resources

About