Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks

Abstract

Responses to natural stimuli in area V4, a mid-level area of the visual ventral stream, are well predicted by features from convolutional neural networks (CNNs) trained on image classification. This result has been taken as evidence for the functional role of V4 in object classification. However, we currently do not know if and to what extent V4 plays a role in solving other computational objectives. Here, we investigated normative accounts of V4 by predicting macaque single-neuron responses to natural images from the representations extracted by 23 CNNs trained on different computer vision tasks including semantic, geometric, 2D, and 3D visual tasks. We found that semantic classification tasks do indeed provide the best predictive features for V4. Other tasks (3D in particular) followed very closely in performance, but a similar pattern of tasks performance emerged when predicting the activations of a network exclusively trained on object recognition. Thus, our results support V4's main functional role in semantic processing. At the same time, they suggest that V4's affinity to various 3D and 2D stimulus features found by electrophysiologists could be a corollary of a semantic functional goal.

High-performing neural network models of visual cortex benefit from high latent dimensionality

“…Our finding that top-performing encoding models of visual cortex tend to have high dimensionality was surprising given that previous work has either not considered latent dimensionality (Cadena et al, 2022; Cao & Yamins, 2021a, 2021b; Conwell et al, 2022; Dwivedi et al, 2021; Khaligh-Razavi & Kriegeskorte, 2014; Konkle & Alvarez, 2022; Kriegeskorte, 2015; Lindsay, 2020; Yamins & DiCarlo, 2016; Yamins et al, 2014; Zhuang et al, 2021) or argued for the opposite of what we discovered: namely that low-dimensional representations better account for biological vision and exhibit computational benefits in terms of robustness and categorization performance (Ansuini et al, 2019; Cohen et al, 2020; Lehky et al, 2014). We wondered whether there might be some important computational benefits of high-dimensional manifolds that have been largely missed in the previous literature.…”

“…Existing hypotheses for the success of deep learning models of visual cortex include the training task, training data, architecture, and layer depth (e.g. Cadena et al, 2022; Cao & Yamins, 2021a, 2021b; Conwell et al, 2022; Dwivedi et al, 2021; Khaligh-Razavi & Kriegeskorte, 2014; Konkle & Alvarez, 2022; Kriegeskorte, 2015; Lindsay, 2020; Yamins & DiCarlo, 2016; Yamins et al, 2014; Zhuang et al, 2021). We therefore examined a large bank of 536 DNN encoding models that vary across each of these factors.…”

“…This perspective implicitly underlies most deep learning approaches for modeling visual cortex, which emphasize factors affecting the alignment between a model and the brain, such as architecture, layer depth, learning objective, and training images (e.g. Cadena et al, 2022; Cao & Yamins, 2021a, 2021b; Conwell et al, 2022; Dwivedi et al, 2021; Khaligh-Razavi & Kriegeskorte, 2014; Konkle & Alvarez, 2022; Kriegeskorte, 2015; Lindsay, 2020; Yamins & DiCarlo, 2016; Yamins et al, 2014; Zhuang et al, 2021). The alignment-based perspective does not entail any specific predictions about ED and, thus, suggests the null hypothesis that ED and encoding scores are unrelated (Figure 1c left panel).…”

“…Our findings also have implications for how neuroscientists interpret the relationship between DNNs and neural representations. When developing theories to explain why some DNNs are better than others as computational brain models, it is common for researchers to place a strong emphasis on the role of design factors, such as architecture, training data, and learning objective (Cadena et al, 2022; Cao & Yamins, 2021a, 2021b; Conwell et al, 2022; Dwivedi et al, 2021; Khaligh-Razavi & Kriegeskorte, 2014; Konkle & Alvarez, 2022; Kriegeskorte, 2015; Lindsay, 2020; Yamins & DiCarlo, 2016; Yamins et al, 2014; Zhuang et al, 2021). However, our findings suggest that these design factors may be of secondary importance, with geometric factors best explaining the relationship between DNNs and neural representations.…”

“…When using DNNs to model neural systems, one of the fundamental questions that researchers hope to answer is: What core factors explain why some DNNs succeed and others fail? Researchers often attribute the success of DNNs to explicit design choices in a model’s construction, such as its architecture, learning objective, and training data (Cadena et al, 2022; Cao & Yamins, 2021a, 2021b; Conwell, Prince, Alvarez, & Konkle, 2022; Dwivedi, Bonner, Cichy, & Roig, 2021; Khaligh-Razavi & Kriegeskorte, 2014; Konkle & Alvarez, 2022; Kriegeskorte, 2015; Lindsay, 2020; Yamins & DiCarlo, 2016; Yamins et al, 2014; Zhuang et al, 2021). However, an alternative perspective explains DNNs through the geometry of their latent representational subspaces, which abstracts over the details of training procedures and architectures (Chung & Abbott, 2021; Chung, Lee, & Sompolinsky, 2018; Cohen, Chung, Lee, & Sompolinsky, 2020; Jazayeri & Ostojic, 2021; Sorscher, Ganguli, & Sompolinsky, 2021).…”