Efficient neural codes naturally emerge through gradient descent learning

Benjamin, Ari S.; Zhang, Ling-Qi; Qiu, Cheng; Stocker, Alan A.; Körding, Konrad P.

doi:10.1038/s41467-022-35659-7

Cited by 18 publications

(14 citation statements)

References 64 publications

(85 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This explains why networks trained with different random initializations end up with similar high-variance principal components but different low-variance components, and it explains why measures of network similarity appear to be more meaningful when they are weighted by variance [51, 56]. Furthermore, these empirical ob-servations are consistent with theories of deep learning which argue that networks progressively learn the principal components of their training data and tasks, with the number of learned components increasing as a function of task and data complexity [41, 42, 36, 7, 76, 56].…”

Section: Resultsmentioning

confidence: 56%

High-performing neural network models of visual cortex benefit from high latent dimensionality

Elmoznino

Bonner

2022

Preprint

View full text Add to dashboard Cite

Geometric descriptions of deep neural networks (DNNs) have the potential to uncover core principles of computational models in neuroscience, while abstracting over the details of model architectures and training paradigms. Here we examined the geometry of DNN models of visual cortex by quantifying the latent dimensionality of their natural image representations. The prevailing view holds that optimal DNNs compress their representations onto low-dimensional manifolds to achieve invariance and robustness, which suggests that better models of visual cortex should have low-dimensional geometries. Surprisingly, we found a strong trend in the opposite direction---neural networks with high-dimensional image manifolds tend to have better generalization performance when predicting cortical responses to held-out stimuli in both monkey electrophysiology and human fMRI data. These findings held across a diversity of design parameters for DNNs, and they suggest a general principle whereby high-dimensional geometry confers a striking benefit to DNN models of visual cortex.

show abstract

Section: Resultsmentioning

confidence: 56%

High-performing neural network models of visual cortex benefit from high latent dimensionality

Elmoznino

Bonner

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For instance, if the visual system more frequently takes horizontal and vertical orientations as input, it could become more sensitive to those orientations. Thus, the visual system could learn to 'efficiently encode' information in a way that would result in the exact sorts of oblique biases we have described here (see, e.g., Benjamin et al, 2022). If this is truethat these effects arise from the input of the natural environment -it remains unclear why these biases would manifest in modalities separate from that input.…”

Section: Interim Discussionmentioning

confidence: 86%

Oblique biases: An instance of domain- and modality-general spatial representation

Yousif¹,

McDougle²

2023

Preprint

View full text Add to dashboard Cite

A variety of phenomena related to the oblique regions of space have been observed across modality (e.g., in vision and in action) and across domain (e.g., for properties like orientation and location). For instance, the classic ‘oblique effect’ describes a deficit in visual acuity for oriented lines in the oblique regions of space, and classic ‘prototype effects’ describe a bias to mis-localize objects towards the oblique regions of space. While there has been speculation that some ‘oblique-related effects’ share a common mechanism, many of these effects are explained in very different terms. The oblique effect itself is often understood as arising from coding asymmetries in orientation-selective neurons in the brain, whereas prototype effects have been described as arising from categorical biases in higher-level cognition. But is it mere coincidence that there are so many distinct effects linked to the oblique regions of space? Here, we explore the possibility that most, if not all, known ‘oblique-related effects’ may stem from a single, underlying spatial representation. In two first experiments, we show that individuals show stable oblique biases across domain and across modality, and we explore how both biases may have a common cause. Then, in a final experiment, we show that this perspective correctly predicts behavior in a novel spatial judgment task. Thus, we argue that a single (distorted) spatial representation may be the root cause of dozens of known phenomena.

show abstract

“…Previous studies have shown that deep neural networks implicitly learn to encode stimulus features as predicted by efficient coding (Benjamin et al, 2022). Here we use "PredNet", a recurrent neural network designed and trained to predict the next frame in a video sequence (Lotter et al, 2016), to test to what degree sensory representations can dynamically change depending on the temporal input statistics.…”

Section: Predictive Neural Networkmentioning

confidence: 99%

Adaptation optimizes sensory encoding of future stimuli

Mao,

Rothkopf,

Stocker

2024

Preprint

Self Cite

View full text Add to dashboard Cite

Sensory neurons continually adapt their response characteristics according to recent sensory input. However, it is unclear how such a reactive process shaped by sensory history can benefit the organism going forward. Here, we test the hypothesis that adaptation indeed acts proactively in the sense that it optimally adjusts sensory encoding for the future, i.e. for the next expected sensory input. We first quantified adaptation induced changes in sensory encoding by psychophysically measuring discrimination thresholds for visual orientation under different adaptation conditions. Using an information theoretic analysis, we found that adaptation consistently reallocates coding resources such that encoding accuracy peaks at the adaptor orientation while total coding capacity remains constant. We then asked whether this characteristic change in encoding accuracy is predicted by the temporal statistics of natural visual input. By analyzing the retinal input of freely behaving human subjects in natural environments, we found that the distribution of local visual orientations in the retinal input stream at any moment in time is also peaked at the mean orientation computed over a short input history leading up to that moment. We further tested our hypothesis with a recurrent neural network trained to predict the next frame of natural scene videos (PredNet). We simulated our human adaptation experiment with PredNet while analyzing its internal sensory representation. We found that the network exhibited the same change in encoding accuracy as observed in human subjects, and as predicted by the natural input statistics. Taken together, our results suggest that adaptation induced changes in encoding accuracy are an attempt of the visual systems to be best possibly prepared for future sensory input.

show abstract

Efficient neural codes naturally emerge through gradient descent learning

Cited by 18 publications

References 64 publications

High-performing neural network models of visual cortex benefit from high latent dimensionality

High-performing neural network models of visual cortex benefit from high latent dimensionality

Oblique biases: An instance of domain- and modality-general spatial representation

Adaptation optimizes sensory encoding of future stimuli

Contact Info

Product

Resources

About