From photos to sketches - how humans and deep neural networks process objects across different levels of visual abstraction

Abstract

Line drawings convey meaning with just a few strokes. Despite strong simplifications, humans can recognize objects depicted in such abstracted images without effort. To what degree do deep convolutional neural networks (CNNs) mirror this human ability to generalize to abstracted object images? While CNNs trained on natural images have been shown to exhibit poor classification performance on drawings, other work has demonstrated highly similar latent representations in the networks for abstracted and natural images. Here, we address these seemingly conflicting findings by analyzing the activation patterns of a CNN trained on natural images across a set of photographs, drawings, and sketches of the same objects and comparing them to human behavior. We find a highly similar representational structure across levels of visual abstraction in early and intermediate layers of the network. This similarity, however, does not translate to later stages in the network, resulting in low classification performance for drawings and sketches. We identified that texture bias in CNNs contributes to the dissimilar representational structure in late layers and the poor performance on drawings. Finally, by fine-tuning late network layers with object drawings, we show that performance can be largely restored, demonstrating the general utility of features learned on natural images in early and intermediate layers for the recognition of drawings. In conclusion, generalization to abstracted images, such as drawings, seems to be an emergent property of CNNs trained on natural images, which is, however, suppressed by domain-related biases that arise during later processing stages in the network.

The spatiotemporal neural dynamics of object recognition for natural images and line drawings

“…This suggests that object recognition can be resolved with the same amount of processing resources for different levels of visual abstraction of the image. This is consistent with previous computational work showing that representations for photographs and drawings at different levels of visual abstraction become highly similar when being processed in feedforward deep convolutional neural networks trained to categorize natural object images (Fan et al, 2018;Singer et al, 2022). While other work has demonstrated that additional recurrent processing is necessary for resolving degraded (Wyatte et al, 2012), occluded (Rajaei et al, 2019;Tang et al, 2018) or otherwise challenging images (Kar et al, 2019), our findings indicate that no additional mechanisms are needed for the robust recognition of abstract drawings.…”

“…As expected, photos and drawings showed the highest RDM correlation (r=0.79) while the correlation for photos and sketches (r=0.41) as well as the correlation between drawings and sketches (r=0.45) were lower. Next, to confirm that human subjects perceive the object images in the different types of depiction similarly at a conceptual level, we used previously acquired data (Singer et al, 2022) where workers on Amazon Mechanical Turk indicated which of three object images they thought was the odd-one out (Hebart et al, 2020). These triplet judgments were used to construct perceptual similarity matrices for each type of depiction separately, which we subsequently correlated to each other to estimate their representational similarity.…”

The Spatiotemporal Neural Dynamics of Object Recognition for Natural Images and Line Drawings

“…This suggests that object recognition can be resolved with the same amount of processing resources for different levels of visual abstraction of the image. This is consistent with previous computational work showing that representations for photographs and drawings at different levels of visual abstraction become highly similar when being processed in feedforward deep convolutional neural networks trained to categorize natural object images ( Fan et al, 2018 ; Singer et al, 2022 ). While other work has demonstrated that additional recurrent processing is necessary for resolving degraded ( Wyatte et al, 2012 ), occluded ( Tang et al, 2018 ; Rajaei et al, 2019 ), or otherwise challenging images ( Kar et al, 2019 ), our findings indicate that no additional mechanisms are needed for the robust recognition of abstract drawings.…”

“…As expected, photographs and drawings showed the highest RDM correlation ( r = 0.79) while the correlation for photographs and sketches ( r = 0.41) as well as the correlation between drawings and sketches ( r = 0.45) were lower. Next, to confirm that human subjects perceive the object images in the different types of depiction similarly at a conceptual level, we used previously acquired data ( Singer et al, 2022 ) where workers on Amazon Mechanical Turk indicated which of three object images they thought was the odd-one-out ( Hebart et al, 2020 ). These triplet judgments were used to construct perceptual similarity matrices for each type of depiction separately, which we subsequently correlated to each other to estimate their representational similarity.…”

“…To ensure that human participants perceive the stimuli in the different types of depiction similarly at a conceptual level, we used data from a previous study ( Singer et al, 2022 ) in which workers on Amazon Mechanical Turk had performed a triplet odd-one-out task ( Hebart et al, 2020 ) on the same stimuli as used here. In this task, participants were instructed to find the odd-one-out in triplets of object images belonging to the same type of depiction.…”