Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity

Abstract

Convolutional neural networks (CNNs) trained for object recognition have been widely used to account for visually-driven neural responses in both the human and primate brains. However, because of the generality and complexity of the task of object classification, it is often difficult to make precise inferences about neural information processing using CNN representations from object classification despite the fact that these representations are effective for predicting brain activity. To better understand underlying the nature of the visual features encoded in different brain regions of the human brain, we predicted brain responses to images using fine-grained representations drawn from 19 specific computer vision tasks. Individual encoding models for each task were constructed and then applied to BOLD5000-a large-scale dataset comprised of fMRI scans collected while observers viewed over 5000 naturalistic scene and object images. Because different encoding models predict activity in different brain regions, we were able to associate specific vision tasks with each region. For example, within scene-selective brain regions, features from 3D tasks such as 3D keypoints and 3D edges explain greater variance as compared to 2D tasks-a pattern that replicates across the whole brain. Using results across all 19 task representations, we constructed a "task graph" based on the spatial layout of well-predicted brain areas from each task. We then compared the brain-derived task structure with the task structure derived from transfer learning accuracy in order to assess the degree of shared information between the two task spaces. These computationally-driven results-arising out of state-of-the-art computer vision methods-begin to reveal the task-specific architecture of the human visual system.

A massive 7T fMRI dataset to bridge cognitive and computational neuroscience

“…The availability of NSD thus opens the door to using brain activity to directly guide the optimization of deep neural networks. Brain-optimized networks (Seeliger et al, 2021) are a useful alternative to task-optimized networks (Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al, 2014), as the narrowly defined tasks that such networks are typically trained to solve do not necessarily respect the diversity of functions supported by the human visual system (Wang et al, 2019) nor necessarily match properties found in biological visual systems (Sinz et al, 2019). In this regard, NSD is a resource that will contribute to a virtuous cycle of competition between models derived from biological and artificial intelligence.…”

Unveiling functions of the visual cortex using task-specific deep neural networks

“…Our study provides a systematic and comprehensive picture of human brain functions using DNNs trained on different tasks. Previous studies 3,4,6,7,17-21 have compared model performance in explaining brain activity, but were limited to a few preselected regions and models, or had a different goal (comparing task structure) 22 . We Figure 2b).…”

“…Our study provides a systematic and comprehensive picture of human brain functions using DNNs trained on different tasks. Previous studies 3,4,6,7,17–21 have compared model performance in explaining brain activity, but were limited to a few preselected regions and models, or had a different goal (comparing task structure) 22 . We go beyond these efforts by comparing fMRI responses across the whole visual brain using a larger set of DNNs, providing a comprehensive account of the function of human visual brain regions.…”

“…Our study provides a systematic and comprehensive picture of human brain functions using DNNs trained on different tasks. Previous studies 3,4,6,7,17-21,51,52 have compared model performance in explaining brain activity, but were limited to a few preselected regions and models, or had a different goal (comparing task structure) 22 . Using the same fMRI dataset as used in this study, a previous study 17 showed that representation in scene-selective ROIs consists of both location and category information using scene-parsing DNNs.…”

Neural Regression, Representational Similarity, Model Zoology & Neural Taskonomy at Scale in Rodent Visual Cortex

“…Key to the engineering of Taskonomy is the use of an encoder-decoder design in which only the construction of the decoder varies across tasks. While recent analyses using a similar approach in human visual cortex with fMRI data [52] have tended to focus only on the latent space of each task's encoder, we choose to extract representations across all layers, better situating Taskonomy within the same empirical paradigm that has so far defined the modeling of object recognition in the primate brain.…”