Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks

Abstract

Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNN models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNN models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNN models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models. Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.

Shape from Contour: Computation and Representation

“…Most intriguingly, this tuning tended to be monotonic rather than bell-shaped: The preferred stimulus for a neuron tended to be at the extreme end of the deformation range. This is quite different from prototype models of population coding for shape that assume that each neuron will have bell-shaped tuning around a preferred stimulus and that these preferred stimuli will be broadly distributed across the parameter space (e.g., Riesenhuber & Poggio 2002). Such bell-shaped tuning for 3D pose has been observed in IT for 3D objects (Logothetis et al 1995); however, variation in 3D pose results in discontinuous configural changes in the object projection (i.e., the occlusion or dis-occlusion of visual features) that form a natural partitioning of the view sphere into prototypes.…”

Decoding and mapping task states of the human brain via deep learning

“…Analogous to the brain's sensory network, DNNs perform complex computations through deep stacks of simple intra-layer neural circuits. Thus, researchers have widely used DNN models to understand the human brain network, especially sensory brain networks (Eickenberg, Gramfort, Varoquaux, & Thirion, 2017;Guclu & van Gerven, 2015;Horikawa & Kamitani, 2017;Rajalingham et al, 2018;Yamins & DiCarlo, 2016). At the same time, DNNs are capable of discovering complex structures within high-dimensional input data, and can transform these structures into abstract levels (LeCun et al, 2015).…”

Bridging the gap between artificial vision and touch

“…In addition to providing experimental evidence of well-studied principles that underlie human grasping, this data-driven exploration could improve our understanding of the function of touch during object manipulation. Deep-learning models have greatly advanced our knowledge of the neural mechanisms that underlie visual object recognition 3 . In this respect, a similar approach could be applied to the interpretation of tactile-information processing in the brain.…”