Topographic Deep Artificial Neural Networks (TDANNs) predict face selectivity topography in primate inferior temporal (IT) cortex

Lee, Hyodong; DiCarlo, James J.

doi:10.32470/ccn.2018.1085-0

Cited by 5 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, aside from the general targeting of the fronto-temporal language network, it is unclear which parts of a model map onto which components of the brain’s language processing mechanisms. In models of vision, for instance, attempts are made to map ANN layers and neurons onto cortical regions (Kubilius et al, 2019) and sub-regions (Lee & DiCarlo, 2018). However, whereas function and its mapping onto anatomy is at least coarsely defined in the case of vision (Felleman & Van Essen, 1991), a similar mapping is not yet established in language beyond the broad distinction between perceptual processing and higher-level linguistic interpretation (e.g.…”

Section: Discussionmentioning

confidence: 99%

The neural architecture of language: Integrative modeling converges on predictive processing

Schrimpf

Blank²,

Tuckute

et al. 2020

Preprint

View full text Add to dashboard Cite

12brain predictivity, with each model's untrained score predictive of its trained score. These results support the hypothesis that a 13 drive to predict future inputs may shape human language processing, and perhaps the way knowledge of language is learned 14 and organized in the brain. In addition, the finding of strong correspondences between ANNs and human representations opens 15 the door to using the growing suite of tools for neural network interpretation to test hypotheses about the human mind. 16 computational neuroscience, language comprehension, fMRI, ECoG, human brain recordings, natural language processing, artificial neural networks, deep learning Specific models accurately predict human brain activity. We found ( Fig. 2a-b) that specific models predict Pereira2018 and 114Fedorenko2016 datasets with up to 100% predictivity (see Fig. S2 for generalization to another metric) relative to the noise 115 ceiling (Methods-7, Fig. S1). The Blank2014 dataset is also reliably predicted, but with lower predictivity. Models vary 116 substantially in their ability to predict neural data. Generally, embedding models such as GloVe do not perform well on any 117 dataset. In contrast, recurrent networks such as skip-thoughts, as well as transformers such as BERT, predict large portions 118 of the data. The model that predicts the human data best across datasets is GPT2-xl, which predicts Pereira2018 and 119Fedorenko2016 at close to 100% and is among the highest-performing models on Blank2014 with 32% predictivity. These 120 scores are higher in the language network than other parts of the brain (SI-4). 121Model scores are consistent across experiments/datasets. To test the generality of the model representations, we examined the 122 consistency of model scores across datasets. Indeed, if a model does well on one dataset, it tends to also do well on other 123 datasets ( Fig. 2c), ruling out the possibility that we are picking up on spurious, dataset-idiosyncratic predictivity, and 124suggesting that the models' internal representations are general enough to capture brain responses to diverse linguistic 125 materials presented visually or auditorily, and across three independent sets of participants. Specifically, model scores 126across the two experiments in Pereira2018 (overlapping sets of participants) correlate at r=.94 (Pearson here and 127 elsewhere, p<<.00001), scores from Pereira2018 and Fedorenko2016 correlate at r=.50 (p<.001), and from Pereira2018 and 128Blank2014 at r=.63 (p<.0001). 129 130Next-word-prediction task performance predicts neural scores. In vision, ANNs that perform better on the specific task of 131 visual classification also tend to better predict responses in the primate ventral stream (Schrimpf et al., 2018; Yamins et al., 132 2014). Building on work that has established a core role for predictive processing in language (Hale, 2001; Levy, 2008a; 133 Smith & Levy, 2013) and recent findings that ANNs that perform well on a next-word prediction task (a normative task 134 known as 'languag...

show abstract

Section: Discussionmentioning

confidence: 99%

The neural architecture of language: Integrative modeling converges on predictive processing

Schrimpf

Blank²,

Tuckute

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Instead, the networks that most successfully capture primate behavior were those that were additionally constrained to perform simulations. This approach may be understood as a generalization of simultaneously optimizing on multiple tasks (Yang, Cole, and Rajan 2019; Yang et al 2019), or of optimizing for specific tasks in the face of specific regularization (Sussillo et al 2015; Lee and DiCarlo 2019), with the goal of building interpretable models of behavioral and neural phenomena (Saxe, Nelli, and Summerfield 2021). To this end, our work highlights a novel and general approach for testing hypotheses about specific inductive biases that govern human cognition by directly comparing models that do or do not implement those biases.…”

Section: Discussionmentioning

confidence: 99%

The role of mental simulation in primate physical inference abilities

Rajalingham

Piccato

Jazayeri

2021

Preprint

View full text Add to dashboard Cite

Primates can richly parse sensory inputs to infer latent information, and adjust their behavior accordingly. It has been hypothesized that such flexible inferences are aided by simulations of internal models of the external world. However, evidence supporting this hypothesis has been based on behavioral models that do not emulate neural computations. Here, we test this hypothesis by directly comparing the behavior of humans and monkeys in a ball interception task to that of recurrent neural network (RNN) models with or without the capacity to “simulate” the underlying latent variables. Humans and monkeys had strikingly similar behavioral patterns suggesting common underlying neural computations. Comparison between primates and a large class of RNNs revealed that only RNNs that were optimized to simulate the position of the ball were able to accurately capture key features of the behavior such as systematic biases in the inference process. These results are consistent with the hypothesis that primates use mental simulation to make flexible inferences. Moreover, our work highlights a general strategy for using model neural systems to test computational hypotheses of higher brain function.

show abstract

“…In particular, aside from the general targeting of the frontotemporal language network, it is unclear which parts of a model map onto which components of the brain's language-processing mechanisms. In models of vision, for instance, attempts are made to map ANN layers and neurons onto cortical regions (3) and subregions (148). However, whereas function and its mapping onto anatomy is at least coarsely defined in the case of vision (149), a similar mapping is not yet established in language beyond the broad distinction between perceptual processing and higherlevel linguistic interpretation (e.g., ref.…”

mentioning

confidence: 99%

The neural architecture of language: Integrative modeling converges on predictive processing

Schrimpf

Blank

Tuckute

et al. 2021

Proc. Natl. Acad. Sci. U.S.A.

291

331

View full text Add to dashboard Cite

The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a systematic study taking this approach to higher-level cognition: human language processing, our species’ signature cognitive skill. We find that the most powerful “transformer” models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (functional MRI and electrocorticography). Models’ neural fits (“brain score”) and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.

show abstract

Topographic Deep Artificial Neural Networks (TDANNs) predict face selectivity topography in primate inferior temporal (IT) cortex

Cited by 5 publications

References 6 publications

The neural architecture of language: Integrative modeling converges on predictive processing

The neural architecture of language: Integrative modeling converges on predictive processing

The role of mental simulation in primate physical inference abilities

The neural architecture of language: Integrative modeling converges on predictive processing

Contact Info

Product

Resources

About