A self-supervised domain-general learning framework for human ventral stream representation

Konkle, Talia; Alvarez, George A.

doi:10.1038/s41467-022-28091-4

Cited by 76 publications

(73 citation statements)

References 85 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although correlations in the face-selective regions were significantly higher than the non-face-selective regions in both the peak intermediate layer and the final fully-connected layer, correlations with the peak intermediate layer were more than five times stronger than with the final fully-connected layer across face-selective regions. An additional analysis excluded the possibility that the low correlation was due to RSA’s inherent assumption of equal weights or scales for all features comprising the two RDMs (Conwell et al, 2021; Kaniuth and Hebart, 2021; Khaligh-Razavi et al, 2017; Konkle and Alvarez, 2022) (see Material and Methods for details and Figure S4 for more results).…”

Section: Resultsmentioning

confidence: 99%

“…Because dynamic information plays a major role in the geometry of brain representations (Haxby et al, 2020b;Nastase et al, 2017;Russ and Leopold, 2015), static images could generate higher correlation values between brain responses and DCNNs that do not use motion information (Daube et al, 2021;Grossman et al, 2019;Tsantani et al, 2021). Similarly, studies that used stimuli spanning superordinate categories (e.g., with multiple visual categories (Konkle and Alvarez, 2022;Murty et al, 2021)) would bias representations towards categorical information, reducing the contribution of information that is needed for within-class individuation such as face identification.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Modeling naturalistic face processing in humans with deep convolutional neural networks

Jiahui

Feilong

Castello

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The relationships between internal representations learned by DCNNs and those of the primate face processing system are not well understood, especially in naturalistic settings. We developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces) and used representational similarity analysis to investigate how well the representations learned by high-performing DCNNs match human brain representations across the entire distributed face processing system. DCNN representational geometries were strikingly consistent across diverse architectures and captured meaningful variance among faces. Similarly, representational geometries throughout the human face network were highly consistent across subjects. Nonetheless, correlations between DCNN and neural representations were very weak overall—DCNNs captured 3% of variance in the neural representational geometries at best. Intermediate DCNN layers better matched visual and face-selective cortices than the final fully-connected layers. Behavioral ratings of face similarity were highly correlated with intermediate layers of DCNNs, but also failed to capture representational geometry in the human brain. Our results suggest that the correspondence between intermediate DCNN layers and neural representations of naturalistic human face processing is weak at best, and diverges even further in the later fully-connected layers. This poor correspondence can be attributed, at least in part, to the dynamic and cognitive information that plays an essential role in human face processing but is not modeled by DCNNs. These mismatches indicate that current DCNNs have limited validity as in silico models of dynamic, naturalistic face processing in humans.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Modeling naturalistic face processing in humans with deep convolutional neural networks

Jiahui

Feilong

Castello

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…One common criticism of task-optimized models is that supervised training on classification tasks is inconsistent with biological learning (45). Recent advances in unsupervised learning have enabled useful representations to be learned from large quantities of natural data without explicit labels, potentially providing a more biologically plausible computational theory of learning (70)(71)(72). A priori it seemed plausible that the invariances of deep neural network models could be strongly dependent on supervised training for classification tasks, in which case models trained without supervision might be substantially more human-like according to the metamers test.…”

Section: Effects Of Unsupervised Trainingmentioning

confidence: 99%

Model metamers illuminate divergences between biological and artificial neural networks

Feather

Leclerc

Mądry

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep neural network models of sensory systems are often proposed to learn representational transformations with invariances like those in the brain. To reveal these invariances we generated “model metamers” – stimuli whose activations within a model stage are matched to those of a natural stimulus. Metamers for state-of-the-art supervised and unsupervised neural network models of vision and audition were often completely unrecognizable to humans when generated from deep model stages, suggesting differences between model and human invariances. Targeted model changes improved human-recognizability of model metamers, but did not eliminate the overall human-model discrepancy. The human-recognizability of a model’s metamers was well predicted by their recognizability by other models, suggesting that models learn idiosyncratic invariances in addition to those required by the task. Metamer recognition dissociated from both traditional brain-based benchmarks and adversarial vulnerability, revealing a distinct failure mode of existing sensory models and providing a complementary benchmark for model assessment.

show abstract

“…Alternatively, the use of more naturalistic “animal-view” movies as input ( Betsch et al, 2004 ), coupled with architectural flexibility, may allow neural networks to learn topographic specializations predictive of patterns in the retina or other hierarchical layers of the visual system ( Doshi and Konkle, 2021 ; Blauch et al, 2022 ). Recent work has shown some progress in this direction, with neural networks explicitly incorporating distinct objectives or constraints that result in the emergence of topographic organization and specializations ( Plaut and Behrmann, 2011 ; Wang and Cottrell, 2017 ; Lee et al, 2020 ; Doshi and Konkle, 2021 ; Zhuang et al, 2021 ; Blauch et al, 2022 ; Konkle and Alvarez, 2022 ), shedding light on the origins of these organizational schemes.…”

Section: Implications For Future Studies Of the Visual Systemmentioning

confidence: 99%

What and Where: Location-Dependent Feature Sensitivity as a Canonical Organizing Principle of the Visual System

Sedigh-Sarvestani

Fitzpatrick

2022

Front. Neural Circuits

View full text Add to dashboard Cite

Traditionally, functional representations in early visual areas are conceived as retinotopic maps preserving ego-centric spatial location information while ensuring that other stimulus features are uniformly represented for all locations in space. Recent results challenge this framework of relatively independent encoding of location and features in the early visual system, emphasizing location-dependent feature sensitivities that reflect specialization of cortical circuits for different locations in visual space. Here we review the evidence for such location-specific encoding including: (1) systematic variation of functional properties within conventional retinotopic maps in the cortex; (2) novel periodic retinotopic transforms that dramatically illustrate the tight linkage of feature sensitivity, spatial location, and cortical circuitry; and (3) retinotopic biases in cortical areas, and groups of areas, that have been defined by their functional specializations. We propose that location-dependent feature sensitivity is a fundamental organizing principle of the visual system that achieves efficient representation of positional regularities in visual experience, and reflects the evolutionary selection of sensory and motor circuits to optimally represent behaviorally relevant information. Future studies are necessary to discover mechanisms underlying joint encoding of location and functional information, how this relates to behavior, emerges during development, and varies across species.

show abstract

A self-supervised domain-general learning framework for human ventral stream representation

Cited by 76 publications

References 85 publications

Modeling naturalistic face processing in humans with deep convolutional neural networks

Modeling naturalistic face processing in humans with deep convolutional neural networks

Model metamers illuminate divergences between biological and artificial neural networks

What and Where: Location-Dependent Feature Sensitivity as a Canonical Organizing Principle of the Visual System

Contact Info

Product

Resources

About