Task-specific vision models explain task-specific areas of visual cortex

Abstract

Computational models such as deep neural networks (DNN) trained for classification are often used to explain responses of the visual cortex. However, not all the areas of the visual cortex are involved in object/scene classification. For instance, scene selective occipital place area (OPA) plays a role in mapping navigational affordances. Therefore, for explaining responses of such task-specific brain area, we investigate if a model that performs a related task can serve as a better computational model than a model that performs an unrelated task. We found that DNN trained on a task (scene-parsing) related to the function (navigational affordances) of a brain region (OPA) explains its responses better than a DNN trained on a task (scene-classification) which is not explicitly related. In a subsequent analysis, we found that the DNNs that showed high correlation with a particular brain region were trained on a task that was consistent with functions of that brain region reported in previous neuroimaging studies. Our results demonstrate that the task is paramount for selecting a computational model of a brain area. Further, explaining the responses of a brain area by a diverse set of tasks has the potential to shed some light on its functions. Author summaryAreas in the human visual cortex are specialized for specific behaviors either due to supervision and interaction with the world or due to evolution. A standard way to gain insight into the function of these brain region is to design experiments related to a particular behavior, and localize the regions showing significant relative activity corresponding to that behavior. In this work, we investigate if we can figure out the function of a brain area in visual cortex using computational vision models. From our results, we find that explaining responses of a brain region using DNNs trained on a diverse set of possible vision tasks can help us gain insights into its function. The consistency of our results using DNNs with the previous neuroimaging studies suggest that the brain region may be specialized for behavior similar to the tasks for which DNNs showed a high correlation with its responses. 2 cortical responses in the visual cortex [1-11]. DNNs trained on a large dataset of images 3 for the object classification task have been shown to explain the human and monkey 4 cortical responses in the inferior temporal cortex (IT) area known for playing a role in 5 August 21, 2018 1/17 object recognition. Further, in Razavi and Kriegsorte [4], it has been revealed that 6unsupervised models are unable to explain the IT responses as well as the models 7 supervised for the classification task, thus emphasizing that supervision for classification 8 task is crucial for explaining IT responses. 9In some recent works [12,13], they show that the fMRI responses in the 10 scene-selective brain regions like OPA and parahippocampal place area (PPA) are 11 correlated with a DNN trained for the classification task. However, in earlier work from 12 Bonner and Epstein [14], they showe...

Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future

“…More ethologically-relevant and embodied tasks such as navigation, object manipulation and visual reasoning, may be needed to capture the full diversity of visual processing and its relation to other brain areas. Early versions of this idea are already being explored [118,119 ]. The study of insect vision has historically taken this more holistic approach and may make for useful inspiration [120].…”

Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity

“…Therefore we can use the predictive regions from each of the models to identify the brain regions where specific task-relevant information is localized. Independently, Dwivedi and Roig [17] has shown that representation similarity analysis (RSA) performed between task representations and brain representations can differentiate scene-selective regions of interest (ROIs) by their preferred task. For example, representations in scene-selective occipital place area (OPA) are more highly correlated with representations from a network trained to predict navigational affordances.…”