The internal representations of early deep artificial neural networks (ANNs) were found to be remarkably similar to the internal neural representations measured experimentally in the primate brain. Here we ask, as deep ANNs have continued to evolve, are they becoming more or less brain-like? ANNs that are most functionally similar to the brain will contain mechanisms that are most like those used by the brain. We therefore developed Brain-Score -a composite of multiple neural and behavioral benchmarks that score any ANN on how similar it is to the brain's mechanisms for core object recognition -and we deployed it to evaluate a wide range of state-of-the-art deep ANNs. Using this scoring system, we here report that: (1) DenseNet-169, CORnet-S and ResNet-101 are the most brain-like ANNs.(2) There remains considerable variability in neural and behavioral responses that is not predicted by any ANN, suggesting that no ANN model has yet captured all the relevant mechanisms.(3) Extending prior work, we found that gains in ANN ImageNet performance led to gains on Brain-Score. However, correlation weakened at ≥ 70% top-1 ImageNet performance, suggesting that additional guidance from neuroscience is needed to make further advances in capturing brain mechanisms. (4) We uncovered smaller (i.e. less complex) ANNs that are more brain-like than many of the best-performing ImageNet models, which suggests the opportunity to simplify ANNs to better understand the ventral stream. The scoring system used here is far from complete. However, we propose that evaluating and tracking model-benchmark correspondences through a Brain-Score that is regularly updated with new brain data is an exciting opportunity: experimental benchmarks can be used to guide machine network evolution, and machine networks are mechanistic hypotheses of the brain's network and thus drive next experiments. To facilitate both of these, we release Brain-Score.org: a platform that hosts the neural and behavioral benchmarks, where ANNs for visual processing can be submitted to receive a Brain-Score and their rank relative to other models, and where new experimental data can be naturally incorporated. computational neuroscience | object recognition | deep neural networks

Particular deep artificial neural networks (ANNs) are today’s most accurate models of the primate brain’s ventral visual stream. Using an ANN-driven image synthesis method, we found that luminous power patterns (i.e., images) can be applied to primate retinae to predictably push the spiking activity of targeted V4 neural sites beyond naturally occurring levels. This method, although not yet perfect, achieves unprecedented independent control of the activity state of entire populations of V4 neural sites, even those with overlapping receptive fields. These results show how the knowledge embedded in today’s ANN models might be used to noninvasively set desired internal brain states at neuron-level resolution, and suggest that more accurate ANN models would produce even more accurate control.

242 words 6Significance Statement 97 words 2 7

Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNN models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNN models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNN models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models. Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.

Particular deep artificial neural networks (ANNs) are today's most accurate models of the primate brain's ventral visual stream. Here we report that, using a targeted ANN-driven image synthesis method, new luminous power patterns (i.e. images) can be applied to the primate retinae to predictably push the spiking activity of targeted V4 neural sites beyond naturally occurring levels. More importantly, this method, while not yet perfect, already achieves unprecedented independent control of the activity state of entire populations of V4 neural sites, even those with overlapping receptive fields. These results show how the knowledge embedded in today's ANN models might be used to non-invasively set desired internal brain states at neuron-level resolution, and suggest that more accurate ANN models would produce even more accurate control.Particular deep feed-forward artificial neural network models (ANNs) constitute today's most accurate "understanding" of the initial ∼200ms of processing in the primate ventral visual stream and the core object recognition behavior it supports (see (1) for the currently leading models). In particular, visually-evoked internal neural representations of these specific ANNs 1 are remarkably similar to the visually-evoked neural representations in mid-level (area V4) and high-level (area IT) cortical stages of the ventral stream (2, 3), a finding that has been extended to neural representations in visual area V1 (4), to patterns of behavioral performance in core object recognition tasks (5, 6), and to both magnetoencephalography and fMRI measurements from the human ventral visual stream (7,8). Notably, these prior findings of model-to-brain similarity were not curve fits to brain data -they were predictions evaluated using images not previously seen by the ANN models, showing that these models have some generalization of their ability to capture key functional properties of the ventral visual stream.However, at least two important potential limitations of this claim have been raised. First, because the visual processing that is executed by the models is not simple to describe, and the models have only been evaluated in terms of internal functional similarity to the brain (above), perhaps they are more like a copy of, rather than a useful "understanding" of, the ventral stream.Second, because the images to assess similarity were sampled from the same distribution as that used to set the model's internal parameters (photograph and rendered object databases), it is unclear if these models would pass a stronger test of functional similarity -does that similarity generalize to entirely novel images? That is, perhaps their reported apparent functional similarity to the brain (3,7,9), substantially over-estimates their true functional similarity.Here we conducted a set of non-human primate visual neurophysiology experiments to assess the first potential limitation by asking if the detailed knowledge that the models contain is useful for one potential application (neural activity control), and t...

We investigated the effect of memory load on encoding and maintenance of information in working memory. Electroencephalography (EEG) signals were recorded while participants performed a modified Sternberg visual memory task. Independent component analysis (ICA) was used to factorise the EEG signals into distinct temporal activations to perform spectrotemporal analysis and localisation of source activities. We found 'encoding' and 'maintenance' operations were correlated with negative and positive changes in α-band power, respectively. Transient activities were observed during encoding of information in the bilateral cuneus, precuneus, inferior parietal gyrus and fusiform gyrus, and a sustained activity in the inferior frontal gyrus. Strong correlations were also observed between changes in α-power and behavioral performance during both encoding and maintenance. Furthermore, it was also found that individuals with higher working memory capacity experienced stronger neural oscillatory responses during the encoding of visual objects into working memory. Our results suggest an interplay between two distinct neural pathways and different spatiotemporal operations during the encoding and maintenance of information which predict individual differences in working memory capacity observed at the behavioral level.

Cortical regions apparently selective to faces, places, and bodies have provided important evidence for domain-specific theories of human cognition, development, and evolution. But claims of category selectivity are not quantitatively precise and remain vulnerable to empirical refutation. Here we develop artificial neural network-based encoding models that accurately predict the response to novel images in the fusiform face area, parahippocampal place area, and extrastriate body area, outperforming descriptive models and experts. We use these models to subject claims of category selectivity to strong tests, by screening for and synthesizing images predicted to produce high responses. We find that these high-response-predicted images are all unambiguous members of the hypothesized preferred category for each region. These results provide accurate, image-computable encoding models of each category-selective region, strengthen evidence for domain specificity in the brain, and point the way for future research characterizing the functional organization of the brain with unprecedented computational precision.

Schizophrenia is often associated with disrupted brain connectivity. However, identifying specific neuroimaging-based patterns pathognomonic for schizophrenia and related symptom severity remains a challenging open problem requiring large-scale datadriven analyses emphasizing not only statistical significance but also stability across multiple datasets, contexts and cohorts. Accurate prediction on previously unseen subjects, or generalization, is also essential for any useful biomarker of schizophrenia. In order to build a predictive model based on functional network feature patterns, we studied whole-brain fMRI functional networks, both at the voxel level and lower-resolution supervoxel level. Targeting Auditory Oddball task data on the FBIRN fMRI dataset (n = 95), we considered node-degree and link-weight network features and evaluated stability and generalization accuracy of statistically significant feature sets in discriminating patients vs. controls. We also applied sparse multivariate regression (elastic net) to whole-brain functional connectivity features, for the first time, to derive stable predictive features for symptom severity. Wholebrain link-weight features achieved 74% accuracy in identifying patients and were more stable than voxel-wise node-degrees. Linkweight features predicted severity of several negative and positive symptom scales, including inattentiveness and bizarre behavior. The most-significant, stable and discriminative functional connectivity changes involved increased correlations between thalamus and primary motor/primary sensory cortex, and between precuneus (BA7) and thalamus, putamen, and Brodmann areas BA9 and BA44. Precuneus, along with BA6 and primary sensory cortex, was also involved in predicting severity of several symptoms. Overall, the proposed multi-step methodology may help identify more reliable multivariate patterns allowing for accurate prediction of schizophrenia and its symptoms severity.