Using goal-driven deep learning models to understand sensory cortex

Yamins, Daniel; DiCarlo, James J.

doi:10.1038/nn.4244

Cited by 1,323 publications

(1,278 citation statements)

References 59 publications

Supporting

Mentioning

1,148

Contrasting

Unclassified

Order By: Relevance

“…As others (Kriegeskorte, 2015;Yamins & DiCarlo, 2016), we therefore believe that there is a strong case that DNNs can serve as a model for information processing in the brain. From this perspective, using DNNs to understand the human brain and behavior is similar to using an animal model.…”

Section: 1mentioning

confidence: 54%

Visual pathways from the perspective of cost functions and multi-task deep neural networks

Scholte

Losch

Ramakrishnan

et al. 2018

Cortex

View full text Add to dashboard Cite

Deep learning Cost functions RepresentationsVisual processing a b s t r a c t Vision research has been shaped by the seminal insight that we can understand the higher-tier visual cortex from the perspective of multiple functional pathways with different goals. In this paper, we try to give a computational account of the functional organization of this system by reasoning from the perspective of multi-task deep neural networks. Machine learning has shown that tasks become easier to solve when they are decomposed into subtasks with their own cost function. We hypothesize that the visual system optimizes multiple cost functions of unrelated tasks and this causes the emergence of a ventral pathway dedicated to vision for perception, and a dorsal pathway dedicated to vision for action.To evaluate the functional organization in multi-task deep neural networks, we propose a method that measures the contribution of a unit towards each task, applying it to two networks that have been trained on either two related or two unrelated tasks, using an identical stimulus set. Results show that the network trained on the unrelated tasks shows a decreasing degree of feature representation sharing towards higher-tier layers while the network trained on related tasks uniformly shows high degree of sharing.We conjecture that the method we propose can be used to analyze the anatomical and functional organization of the visual system and beyond. We predict that the degree to which tasks are related is a good descriptor of the degree to which they share downstream cortical-units.

show abstract

Section: 1mentioning

confidence: 54%

Visual pathways from the perspective of cost functions and multi-task deep neural networks

Scholte

Losch

Ramakrishnan

et al. 2018

Cortex

View full text Add to dashboard Cite

show abstract

“…1 However, when deep or recurrent neural networks are used to model neurobiological systems, the comparison between model activity and brain activity is often only verified at a coarse resolution, at the level of entire population dynamics 2,3 , or linear combinations of neurons [4][5][6] , and in contexts that are not very different from the contexts that the networks were originally trained in. Thus, the advent of deep learning as a modeling approach in neuroscience raises two more fundamental unanswered questions.…”

mentioning

confidence: 99%

The dynamic neural code of the retina for natural scenes

Maheswaranathan

Tanaka

et al. 2018

Preprint

View full text Add to dashboard Cite

The normal function of the retina is to convey information about natural visual images. It is this visual environment that has driven evolution, and that is clinically relevant. Yet nearly all of our understanding of the neural computations, biological function, and circuit mechanisms of the retina comes in the context of artificially structured stimuli such as flashing spots, moving bars and white noise. It is fundamentally unclear how these artificial stimuli are related to circuit processes engaged under natural stimuli. A key barrier is the lack of methods for analyzing retinal responses to natural images. We addressed both these issues by applying convolutional neural network models (CNNs) to capture retinal responses to natural scenes. We find that CNN models predict natural scene responses with high accuracy, achieving performance close to the fundamental limits of predictability set by intrinsic cellular variability. Furthermore, individual internal units of the model are highly correlated with actual retinal interneuron responses that were recorded separately and never presented to the model during training. Finally, we find that models fit only to natural scenes, but not white noise, reproduce a range of phenomena previously described using distinct artificial stimuli, including frequency doubling, latency encoding, motion anticipation, fast contrast adaptation, synchronized responses to motion reversal and object motion sensitivity. Further examination of the model revealed extremely rapid context dependence of retinal feature sensitivity under natural scenes using an analysis not feasible from direct examination of retinal responses. Overall, these results show that nonlinear retinal processes engaged by artificial stimuli are also engaged in and relevant to natural visual processing, and that CNN models form a powerful and unifying tool to study how sensory circuitry produces computations in a natural context.

show abstract

“…The current experiment builds on recent work finding a strong correspondence between representations in goal-directed CNNs trained for visual categorization and brain activity [34][35][36] . Unit activity in object categorization CNNs predicts neural firing to natural objects in macaque IT and V4 37 .…”

Section: Testing the Biological Validity Of Cnn Features As A Model Omentioning

confidence: 98%

Predicting eye movements from deep neural network activity decoded from fMRI responses to natural scenes

Chun

2017

Preprint

View full text Add to dashboard Cite

Computational models of selective spatial attention can reliably predict eye movements to complex images. However, researchers lack a simple way to measure covert representations of spatial attention in the brain and their link to overt eye movement behavior, especially in response to natural scenes. Here, we predict eye movement patterns from spatial priority maps reconstructed from brain activity measured with functional magnetic resonance imaging (fMRI). First, we define a computational spatial attention model using a deep convolutional neural network (CNN) pre-trained for scene categorization. Next, we decode CNN unit activity from fMRI activity and reconstruct spatial priority maps by applying our computational spatial attention model to decoded CNN activity. Finally, we predict eye movements in a subsequent behavioral experiment within and between individuals using reconstructed spatial priority maps. These results demonstrate that features represented in CNN unit activity can guide spatial attention and eye movements, providing a crucial link between CNN models, brain activity, and behavior.

show abstract

Using goal-driven deep learning models to understand sensory cortex

Cited by 1,323 publications

References 59 publications

Visual pathways from the perspective of cost functions and multi-task deep neural networks

Visual pathways from the perspective of cost functions and multi-task deep neural networks

The dynamic neural code of the retina for natural scenes

Predicting eye movements from deep neural network activity decoded from fMRI responses to natural scenes

Contact Info

Product

Resources

About