Perceptual straightening of natural videos

Hénaff, Olivier J.; Goris, Robbe L. T.; Simoncelli, Eero P.

doi:10.1038/s41593-019-0377-4

Cited by 61 publications

(61 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1. This intuition underlies a number of studies on object representations in the brain 1,[12][13][14][15] , and in deep artificial neural networks [16][17][18][19] . As separability of manifolds depends on numerous factors-numbers of neurons and manifolds, sizes and shapes of the manifolds, target labels, among others-it has been difficult to elucidate which specific properties of the representation truly contribute to untangling.…”

mentioning

confidence: 99%

“…Another approach, representational similarity analysis (RSA), uses similarity matrices to determine which stimuli are more correlated within neural data and in network representations 4,7,25 . Others have considered various measures such as curvature 15 and dimensionality to capture the structure within neural representations [26][27][28][29][30][31][32] ; but it is unclear how these measures are related to task performance such as classification accuracy. Conversely, others have explored functional aspects by using different layer representations for transfer learning 33 , or object classification 31 , but it is unclear why performance improves or deteriorates.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Separability and geometry of object manifolds in deep neural networks

et al. 2020

View full text Add to dashboard Cite

Stimuli are represented in the brain by the collective population responses of sensory neurons, and an object presented under varying conditions gives rise to a collection of neural population responses called an 'object manifold'. Changes in the object representation along a hierarchical sensory system are associated with changes in the geometry of those manifolds, and recent theoretical progress connects this geometry with 'classification capacity', a quantitative measure of the ability to support object classification. Deep neural networks trained on object classification tasks are a natural testbed for the applicability of this relation. We show how classification capacity improves along the hierarchies of deep neural networks with different architectures. We demonstrate that changes in the geometry of the associated object manifolds underlie this improved capacity, and shed light on the functional roles different levels in the hierarchy play to achieve it, through orchestrated reduction of manifolds' radius, dimensionality and inter-manifold correlations.

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

Separability and geometry of object manifolds in deep neural networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Neurons in an area that is specialized for representing objects should depend less on object parameters that preserve the identity such as position, scale, pose and illumination conditions (DiCarlo and Cox, 2007;Hénaff et al, 2019;Rust and DiCarlo, 2010;Tafazoli et al, 2017) . As a result, an object decoder built on a subset of the nuisance parameter space (for example a limited range of translations, sizes, and rotations) should generalize across nuisance parameters.…”

mentioning

confidence: 99%

Object manifold geometry across the mouse cortical visual hierarchy

Froudarakis

Cohen

Diamantaki

et al. 2020

Preprint

View full text Add to dashboard Cite

Despite variations in appearance we robustly recognize objects. Neuronal populations responding to objects presented under varying conditions form object manifolds and hierarchically organized visual areas are thought to untangle pixel intensities into linearly decodable object representations. However, the associated changes in the geometry of object manifolds along the cortex remain unknown. Using home cage training we showed that mice are capable of invariant object recognition. We simultaneously recorded the responses of thousands of neurons to measure the information about object identity available across the visual cortex and found that lateral visual areas LM, LI and AL carry more linearly decodable object identity information compared to other visual areas. We applied the theory of linear separability of manifolds, and found that the increase in classification capacity is associated with a decrease in the dimension and radius of the object manifold, identifying features of the population code that enable invariant object coding.

show abstract

“…Interestingly, however, the absolute discrimination performance was highest in V1, the lowest visual area, at least at short lags ( Figure 3B). This may seem surprising in light of popular conceptual frameworks advocating for "untangling" (DiCarlo et al 2012) and "straightening" (Hénaff et al 2019) of neural representations along the ventral stream. These frameworks suggest that deeper areas, carrying untangled representations of objects, should support better discrimination by a linear classifier, as demonstrated in several monkey studies (Hong et al 2016;Hung et al 2005;Li et al 2009;Rust and Dicarlo 2010).…”

Section: Discussionmentioning

confidence: 99%

Intrinsic dynamics enhance temporal stability of stimulus representation along rodent visual cortical hierarchies

Piasini

Soltuzu

Muratore

et al. 2019

Preprint

View full text Add to dashboard Cite

Along the ventral stream, cortical representations of brief, static stimuli become gradually more invariant to identity-preserving transformations. In the presence of long, ethologically relevant dynamic stimuli, higher invariance should imply temporally persistent representations at the top of this functional hierarchy. However, such stimuli could engage adaptive and predictive processes, whose impact on neural coding dynamics is unknown. More generally, coding dynamics in the presence of temporally structured stimuli are not understood. By probing the rodent analogue of the ventral stream with movies, we uncovered a hierarchy of temporal scales along this pathway, with deeper areas encoding visual information more persistently. Furthermore, the impact of intrinsic dynamics on the stability of stimulus representations gradually grows along the hierarchy. These results suggest that feedforward computations in the cortical hierarchy build up invariance even for dynamic, temporally structured stimuli, and that intrinsic processing contributes to the stabilization of representations in noisy, changing environments.

show abstract

Perceptual straightening of natural videos

Cited by 61 publications

References 41 publications

Separability and geometry of object manifolds in deep neural networks

Separability and geometry of object manifolds in deep neural networks

Object manifold geometry across the mouse cortical visual hierarchy

Intrinsic dynamics enhance temporal stability of stimulus representation along rodent visual cortical hierarchies

Contact Info

Product

Resources

About