Higher visual areas act like domain-general filters with strong selectivity and functional specialization

Abstract

Investigation of the visual system has mainly relied on a-priori hypotheses to restrict experimental stimuli or models used to analyze experimental data. Hypotheses are an essential part of scientific inquiry, but an exclusively hypothesis-driven approach might lead to confirmation bias towards existing theories and away from novel discoveries not predicted by them. This paper uses a hypothesis-neutral computational approach to study four high-level visual regions of interest (ROIs) selective to faces, places, letters, or body parts. We leverage the unprecedented scale and quality of the Natural Scenes Dataset to constrain neural network models of these ROIs with functional Magnetic Resonance Imaging (fMRI) measurements. We show that using only the stimulus images and the associated activity in an ROI, we are able to train from scratch a neural network that can predict the activity in each voxel of that ROI with an accuracy that beats state-of-the-art models. Moreover, once trained, the ROI-specific networks can reveal what kinds of functional properties emerge spontaneously in their training. Strikingly, despite no category-level supervision, the units in the trained networks act strongly as detectors for semantic concepts like faces or words, thereby providing substantial pieces of evidence for categorical selectivity in these visual areas. Importantly, this selectivity is maintained when training the networks with selective deprivations in the training diet, by excluding images that contain their preferred category. The resulting selectivity in the trained networks strongly suggests that the visual areas do not function as exclusive category detectors but are also sensitive to visual patterns that are typical to their preferred categories, even in the absence of these categories. Finally, we show that our response-optimized networks have distinct functional properties. Together, our findings suggest that response-optimized models combined with model interpretability techniques can serve as a powerful and unifying computational framework for probing the nature of representations and computations in the brain.

Visual object topographic motifs emerge from self-organization of a unified representational space

“…Some theoretical accounts of these regions consider these as independent and unrelated functional modules, implicitly assuming no direct relationship between them ( Kanwisher, 2010; Zeki, 1978 ) . However, the integrated feature space of the deep neural network allows us to consider an alternate hypothesis that face- and scene-selectivity might naturally emerge as different parts of a common encoding space—one whose features are designed to discriminate among all kinds of objects more generally (Konkle & Caramazza, 2013; Bao et al, 2020; Vinken et al, 2022; Prince & Konkle, 2020; Khosla & Wehbe, 2022). If this is the case, these categories would drive responses in a localized part of the feature space, which would emerge as a localized cluster of selective responses in the SOM.…”

“…Relatedly, Huang et al, (2022) have found that information about the real-world size of objects is encoded along the second principal component of the late stages of deep neural networks ( 44 ). Further, Vinken et al, 2022 recently demonstrated that face-selective neurons in IT could be accounted for by the feature tuning learned in these same object-trained deep neural networks (( 45 ), also see ( 42 , 46 , 43 )). Thus, deep neural networks clearly operationalize a multi-dimensional representational encoding space that has information about these well-studied object distinctions.…”

“…This theoretical account of the tuning and topography of the object-selective cortex has been challenging to test, as there were no image-computable feature spaces rich enough to categorize many kinds of objects (Kourtzi & Conner, 2011). However, deep neural networks trained to do many-way object categorization, without any special feature branches set aside for some categories, provide precisely this kind of a unified representational space (Prince & Konkle, 2020; Khosla & Wehbe, 2022). Indeed, recently, Bao et al, (2020) used a late layer of a deep neural network (AlexNet) to operationalize such a unified representational space, proposing that the monkey IT organization can be thought of as a coarse map of this space.…”

Beyond linear regression: mapping models in cognitive neuroscience should align with research goals

“…Finally, some researchers are trying to build accurate encoding models that, in essence, enable simulations of neuroscience experiments [44,45]. This type of modeling is especially important in cases when experimental data are expensive or hard to obtain: with a high-accuracy model of brain responses, a researcher can run thousands of experiments in silico, refine their hypothesis, and then test the critical predictions in vivo.…”

Beyond linear regression: mapping models in cognitive neuroscience should align with research goals

“…Finally, some researchers are trying to build accurate encoding models that, in essence, enable simulations of neuroscience experiments [44, 45]. This type of modeling is especially important in cases when experimental data are expensive or hard to obtain: with a high-accuracy model of brain responses, a researcher can run thousands of experiments in silico , refine their hypothesis, and then test the critical predictions in vivo .…”