Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks

Rajalingham, Rishi; Issa, Elias B.; Bashivan, Pouya; Kar, Kohitij; Schmidt, Kailyn; DiCarlo, James J.

doi:10.1101/240614

Cited by 130 publications

(210 citation statements)

References 39 publications

Supporting

Mentioning

197

Contrasting

Order By: Relevance

“…Analogous to the brain's sensory network, DNNs perform complex computations through deep stacks of simple intra‐layer neural circuits. Thus, researchers have widely used DNN models to understand the human brain network, especially sensory brain networks (Eickenberg, Gramfort, Varoquaux, & Thirion, ; Guclu & van Gerven, ; Horikawa & Kamitani, ; Rajalingham et al, ; Yamins & DiCarlo, ). At the same time, DNNs are capable of discovering complex structures within high‐dimensional input data, and can transform these structures into abstract levels (LeCun et al, ).…”

Section: Discussionmentioning

confidence: 99%

Decoding and mapping task states of the human brain via deep learning

Wang

Liang

Jiang

et al. 2019

Human Brain Mapping

View full text Add to dashboard Cite

Support vector machine (SVM)‐based multivariate pattern analysis (MVPA) has delivered promising performance in decoding specific task states based on functional magnetic resonance imaging (fMRI) of the human brain. Conventionally, the SVM‐MVPA requires careful feature selection/extraction according to expert knowledge. In this study, we propose a deep neural network (DNN) for directly decoding multiple brain task states from fMRI signals of the brain without any burden for feature handcrafts. We trained and tested the DNN classifier using task fMRI data from the Human Connectome Project's S1200 dataset (N = 1,034). In tests to verify its performance, the proposed classification method identified seven tasks with an average accuracy of 93.7%. We also showed the general applicability of the DNN for transfer learning to small datasets (N = 43), a situation encountered in typical neuroscience research. The proposed method achieved an average accuracy of 89.0 and 94.7% on a working memory task and a motor classification task, respectively, higher than the accuracy of 69.2 and 68.6% obtained by the SVM‐MVPA. A network visualization analysis showed that the DNN automatically detected features from areas of the brain related to each task. Without incurring the burden of handcrafting the features, the proposed deep decoding method can classify brain task states highly accurately, and is a powerful tool for fMRI researchers.

show abstract

Section: Discussionmentioning

confidence: 99%

Decoding and mapping task states of the human brain via deep learning

Wang

Liang

Jiang

et al. 2019

Human Brain Mapping

View full text Add to dashboard Cite

show abstract

“…For the macaque face-processing network, our study here could not find a CNN layer corresponding to the intermediate stage (ML). In addition, some recent studies have pointed out potential representational discrepancies between CNN and the ventral stream from behavioral consideration 19,23,[27][28][29] . (See also the related discussion on the 'computational gap' below.)…”

Section: Discussionmentioning

confidence: 99%

CNN explains tuning properties of anterior, but not middle, face-processing areas in macaque IT

Raman¹,

Hosoya²

2019

Preprint

View full text Add to dashboard Cite

Recent computational studies have emphasized layer-wise quantitative similarity between convolutional neural networks (CNNs) and the primate visual ventral stream. However, whether such similarity holds for the face-selective areas, a subsystem of the higher visual cortex, is not clear. Here, we extensively investigate whether CNNs exhibit tuning properties as previously observed in different macaque face areas. While simulating four past experiments on a variety of CNN models, we sought for the model layer that quantitatively matches the multiple tuning properties of each face area. Our results show that higher model layers explain reasonably well the properties of anterior areas, while no layer simultaneously explains the properties of middle areas, consistently across the model variation. Thus, the CNNs may have some similarity with the primate face-processing system in the near-goal representation, but differ in the intermediate computational process, thus requiring a more comprehensive model for understanding the entire system.Recently, the neuroscience community has witnessed the rise of the deep convolution neural network (CNN) 1 , a family of feedforward artificial neural networks, in computational modeling of the primate visual system. CNN models trained for behavioral goals have exhibited remarkable similarity to ventral visual areas in terms of stimulus-response relationship despite that the network itself was not directly optimized to fit neural data. For

show abstract

“…Although these network models are not constrained by experimental data, they have nonetheless been shown to provide an even better fit than earlier models to both behavioral 18,24,25 and electrophysiological 26,27 data (but see Ref. 28). These network architectures now achieve accuracy well beyond those of earlier computational models of the VC and are on par with or better than human accuracy during unspeeded image categorization tasks for both object 29 and face 30 recognition.…”

Section: Introductionmentioning

confidence: 99%

Beyond the feedforward sweep: feedback computations in the visual cortex

Kreiman

Serre

2020

Annals of the New York Academy of Sciences

View full text Add to dashboard Cite

Visual perception involves the rapid formation of a coarse image representation at the onset of visual processing, which is iteratively refined by late computational processes. These early versus late time windows approximately map onto feedforward and feedback processes, respectively. State‐of‐the‐art convolutional neural networks, the main engine behind recent machine vision successes, are feedforward architectures. Their successes and limitations provide critical information regarding which visual tasks can be solved by purely feedforward processes and which require feedback mechanisms. We provide an overview of recent work in cognitive neuroscience and machine vision that highlights the possible role of feedback processes for both visual recognition and beyond. We conclude by discussing important open questions for future research.

show abstract

Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks

Abstract: 242 words 6Significance Statement 97 words 2 7

Cited by 130 publications

References 39 publications

Decoding and mapping task states of the human brain via deep learning

Decoding and mapping task states of the human brain via deep learning

CNN explains tuning properties of anterior, but not middle, face-processing areas in macaque IT

Beyond the feedforward sweep: feedback computations in the visual cortex

Contact Info

Product

Resources

About