A performance-optimized model of neural responses across the ventral visual stream

Seibert, Darren; Dl, Yamins; Ardila, Diego; Hong, Ha; DiCarlo, James J.; Gardner, Jl

doi:10.1101/036475

Cited by 14 publications

(23 citation statements)

References 92 publications

(112 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has been shown that DCNN provides the best model out of a wide range of neuroscientific and computer vision models for the neural representation of visual images in high-level visual cortex of monkeys (Yamins et al, 2014) and humans (Khaligh-Razavi and Kriegeskorte, 2014). Other studies have demonstrated with fMRI a direct correspondence between the hierarchy of the human visual areas and layers of the DCNN (Güçlü and van Gerven, 2015; Eickenberg et al, 2016; Seibert et al, 2016; Cichy et al, 2016b). In sum, the increasing feature complexity of the DCNN corresponds to the increasing feature complexity occurring in visual object recognition in the primate brain (Kriegeskorte, 2015; Yamins and DiCarlo, 2016).…”

Section: Introductionmentioning

confidence: 88%

Activations of Deep Convolutional Neural Network are Aligned with Gamma Band Activity of Human Visual Cortex

Kuzovkin

Vicente

Petton

et al. 2017

Preprint

View full text Add to dashboard Cite

Previous work demonstrated a direct correspondence between the hierarchy of the human visual areas and layers of deep convolutional neural networks (DCNN) trained on visual object recognition. We used DCNNs to investigate which frequency bands correlate with feature transformations of increasing complexity along the ventral visual pathway. By capitalizing on intracranial depth recordings from 100 patients and 11293 electrodes we assessed the alignment between the DCNN and signals at different frequency bands in different time windows. We found that gamma activity, especially in the lower spectrum (30 − 70 Hz), matched the increasing complexity of visual feature representations in the DCNN. These findings show that the activity of the DCNN captures the essential characteristics of biological object recognition not only in space and time, but also in the frequency domain. These results also demonstrate the potential that modern artificial intelligence algorithms have in advancing our understanding of the brain. Studying intracranial depth recordings allowed us to extend pre-8 vious works by assessing when and at which frequency bands the 9 activity of the visual system corresponds to the DCNN. Our key 10 Significance Statement

show abstract

Section: Introductionmentioning

confidence: 88%

Activations of Deep Convolutional Neural Network are Aligned with Gamma Band Activity of Human Visual Cortex

Kuzovkin

Vicente

Petton

et al. 2017

Preprint

View full text Add to dashboard Cite

show abstract

“…Yamins, DiCarlo and colleagues 39showed recently that using deep networks trained on large-scale object recognition as 40 nonlinear feature spaces for neural system identification works remarkably well in higher 41 areas of the ventral stream, such as V4 and IT [32,33]. Other groups have used similar 42 approaches for early cortical areas using fMRI [34][35][36]. However, this approach has not 43 yet been used to model spiking activity of early stages such as V1.…”

mentioning

confidence: 99%

Deep convolutional models improve predictions of macaque V1 responses to natural images

Cadena

Denfield

Walker

et al. 2017

Preprint

104

196

View full text Add to dashboard Cite

Despite great efforts over several decades, our best models of primary visual cortex (V1) still predict neural responses quite poorly when probed with natural stimuli, highlighting our limited understanding of the nonlinear computations in V1. At the same time, recent advances in machine learning have shown that deep neural networks can learn highly nonlinear functions for visual information processing.Two approaches based on deep learning have recently been successfully applied to neural data: transfer learning for predicting neural activity in higher areas of the primate ventral stream and data-driven models to predict retina and V1 neural activity of mice. However, so far there exists no comparison between the two approaches and neither of them has been used to model the early primate visual system. Here, we test the ability of both approaches to predict neural responses to natural images in V1 of awake monkeys. We found that both deep learning approaches outperformed classical linearnonlinear and wavelet-based feature representations building on existing V1 encoding theories. On our dataset, transfer learning and data-driven models performed similarly, while the data-driven model employed a much simpler architecture. Thus, multi-layer CNNs set the new state of the art for predicting neural responses to natural images in primate V1. Having such good predictive in-silico models opens the door for quantitative studies of yet unknown nonlinear computations in V1 without being limited by the available experimental time.

show abstract

“…The optimization and cross-validation procedures for Equation 5 were the exact same as those described in Section 2.9, where the learning rate was set to η = 0.001, the stopping criterion was set to ϵ = 0.001 and four-fold cross-validation was performed. This objective function and optimization procedure is similar to that implemented in Seibert et al (2016).…”

Section: Methodsmentioning

confidence: 99%

“…In human neuroimaging studies, recent research has shown that features represented in the shallow layers of a CNN map to fMRI responses in early visual cortex, and features represented in the deeper layers map to responses from higher visual cortex (Güçlü and van Gerven, 2015; Seibert et al, 2016; Wen et al, 2017). Moreover, it has also been shown that stimulus representations in CNNs can be mapped temporally, where peak correspondence between a neural network and MEG responses occurred at earlier time points for shallow layers and at later time points for deeper layers (Cichy et al, 2016; Seeliger et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

Time-Resolved Correspondences Between Deep Neural Network Layers and EEG Measurements in Object Processing

Kong

Kaneshiro

Yamins

et al. 2019

Preprint

View full text Add to dashboard Cite

AbstractThe ventral visual stream is known to be organized hierarchically, where early visual areas processing simplistic features feed into higher visual areas processing more complex features. Hierarchical convolutional neural networks (CNNs) were largely inspired by this type of brain organization and have been successfully used to model neural responses in different areas of the visual system. In this work, we aim to understand how an instance of these models corresponds to temporal dynamics of human object processing. Using representational similarity analysis (RSA) and various similarity metrics, we compare the model representations with two electroencephalography (EEG) data sets containing responses to a shared set of 72 images. We find that there is a hierarchical relationship between the depth of a layer and the time at which peak correlation with the brain response occurs for certain similarity metrics in both data sets. However, when comparing across layers in the neural network, the correlation onset time did not appear in a strictly hierarchical fashion. We present two additional methods that improve upon the achieved correlations by optimally weighting features from the CNN and show that depending on the similarity metric, deeper layers of the CNN provide a better correspondence than shallow layers to later time points in the EEG responses. However, we do not find that shallow layers provide better correspondences than those of deeper layers to early time points, an observation that violates the hierarchy and is in agreement with the finding from the onset-time analysis. This work makes a first comparison of various response features—including multiple similarity metrics and data sets—with respect to a neural network.

show abstract

A performance-optimized model of neural responses across the ventral visual stream

Cited by 14 publications

References 92 publications

Activations of Deep Convolutional Neural Network are Aligned with Gamma Band Activity of Human Visual Cortex

Activations of Deep Convolutional Neural Network are Aligned with Gamma Band Activity of Human Visual Cortex

Deep convolutional models improve predictions of macaque V1 responses to natural images

Time-Resolved Correspondences Between Deep Neural Network Layers and EEG Measurements in Object Processing

Contact Info

Product

Resources

About