Dynamics of scene representations in the human brain revealed by magnetoencephalography and deep neural networks

Cichy, Radoslaw Martin; Khosla, Aditya; Pantazis, Dimitrios; Oliva, Aude

doi:10.1101/032623

Cited by 52 publications

(73 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…previous work using MEG to correlate neural responses with computer vision features focused mainly on temporal patterns(Cichy, Khosla, Pantazis, & Oliva, 2017;Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016c;Clarke et al, 2014). In line with the standard hierarchical model of primate visual cortex, these prior studies observed an earlyto-late shift from lower-level to higher-level features-a result consistent with the results we present here.…”

supporting

confidence: 91%

“…However, in contrast to our present study, these studies focused on recognizing isolated objects on blank backgrounds (Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016c;Clarke et al, 2014), or on specific properties of scenes, restricting the input to a small set of scene stimuli (Cichy, Khosla, Pantazis, & Oliva, 2017). However, in contrast to our present study, these studies focused on recognizing isolated objects on blank backgrounds (Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016c;Clarke et al, 2014), or on specific properties of scenes, restricting the input to a small set of scene stimuli (Cichy, Khosla, Pantazis, & Oliva, 2017).…”

contrasting

confidence: 70%

“…In particular, CNNs typically incorporate multiple layers with units of increasing receptive field sizes as one moves up along the hierarchy; the first layer takes raw image inputs and the last layer outputs object or scene category labels. Moreover, as alluded to earlier, features extracted from CNNs have been found to share significant representational similarity with the neural representations of objects and scenes at both low levels, as exemplified by early visual regions, and high levels, as exemplified by categoryselective brain regions (Cichy, Khosla, Pantazis, & Oliva, 2017;Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016c;Khaligh-Razavi & Kriegeskorte, 2014;Yamins et al, 2014). After such optimization, the layered structure of the network inherently provides an operational definition of low-level to high-level representation of task-relevant visual information (Zeiler & Fergus, 2014).…”

mentioning

confidence: 93%

See 2 more Smart Citations

Exploring spatiotemporal neural dynamics of the human visual cortex

Yang

Tarr

Aminoff

2019

Human Brain Mapping

View full text Add to dashboard Cite

The human visual cortex is organized in a hierarchical manner. Although previous evidence supporting this hypothesis has been accumulated, specific details regarding the spatiotemporal information flow remain open. Here we present detailed spatiotemporal correlation profiles of neural activity with low‐level and high‐level features derived from an eight‐layer neural network pretrained for object recognition. These correlation profiles indicate an early‐to‐late shift from low‐level features to high‐level features and from low‐level regions to higher‐level regions along the visual hierarchy, consistent with feedforward information flow. Additionally, we computed three sets of features from the low‐ and high‐level features provided by the neural network: object‐category‐relevant low‐level features (the common components between low‐level and high‐level features), low‐level features roughly orthogonal to high‐level features (the residual Layer 1 features), and unique high‐level features that were roughly orthogonal to low‐level features (the residual Layer 7 features). Contrasting the correlation effects of the common components and the residual Layer 1 features, we observed that the early visual cortex (EVC) exhibited a similar amount of correlation with the two feature sets early in time, but in a later time window, the EVC exhibited a higher and longer correlation effect with the common components (i.e., the low‐level object‐category‐relevant features) than with the low‐level residual features—an effect unlikely to arise from purely feedforward information flow. Overall, our results indicate that non‐feedforward processes, for example, top‐down influences from mental representations of categories, may facilitate differentiation between these two types of low‐level features within the EVC.

show abstract

supporting

confidence: 91%

contrasting

confidence: 70%

mentioning

confidence: 93%

See 1 more Smart Citation

Exploring spatiotemporal neural dynamics of the human visual cortex

Yang

Tarr

Aminoff

2019

Human Brain Mapping

View full text Add to dashboard Cite

show abstract

“…Our results also shine new light on the temporal processing cascade during scene perception. Sensitivity to spatial structure emerged after 255 ms of processing, which is only after scene-selective peaks in ERPs (Harel et al, 2016;Sato et al, 1999) 9 and after basic scene attributes are computed (Cichy, Khosla, Pantazis, & Oliva, 2017).…”

Section: Discussionmentioning

confidence: 99%

Cortical sensitivity to natural scene structure

Kaiser

Häberle

Cichy

2019

Human Brain Mapping

View full text Add to dashboard Cite

Natural scenes are inherently structured, with meaningful objects appearing in predictable locations. Human vision is tuned to this structure: When scene structure is purposefully jumbled, perception is strongly impaired. Here, we tested how such perceptual effects are reflected in neural sensitivity to scene structure. During separate fMRI and EEG experiments, participants passively viewed scenes whose spatial structure (i.e., the position of scene parts) and categorical structure (i.e., the content of scene parts) could be intact or jumbled. Using multivariate decoding, we show that spatial (but not categorical) scene structure profoundly impacts on cortical processing: Scene‐selective responses in occipital and parahippocampal cortices (fMRI) and after 255 ms (EEG) accurately differentiated between spatially intact and jumbled scenes. Importantly, this differentiation was more pronounced for upright than for inverted scenes, indicating genuine sensitivity to spatial structure rather than sensitivity to low‐level attributes. Our findings suggest that visual scene analysis is tightly linked to the spatial structure of our natural environments. This link between cortical processing and scene structure may be crucial for rapidly parsing naturalistic visual inputs.

show abstract

“…A CNN is not designed to be biologically plausible, although we may draw a loose analogies between a CNN and the human cortex about their processing stages and representations (VanRullen, 2017). Comparative fMRI studies show that representations in the CNN correlate with emerging visual representations in the human brain (Cichy, Khosla, Pantazis, & Oliva, 2017;Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016). Semantic similarity corresponds to more processed visual representations found in final CNN layers (in our case, fc7) or the inferotemporal cortex in the brain, whereas perceptual similarity is based on low-level representations in lower layers of a CNN or in the visual cortex.…”

Section: Discussionmentioning

confidence: 99%

Visual properties and memorising scenes: Effects of image-space sparseness and uniformity

Lukavský

Děchtěrenko

2017

Atten Percept Psychophys

View full text Add to dashboard Cite

Previous studies have demonstrated that humans have a remarkable capacity to memorise a large number of scenes. The research on memorability has shown that memory performance can be predicted by the content of an image. We explored how remembering an image is affected by the image properties within the context of the reference set, including the extent to which it is different from its neighbours (imagespace sparseness) and if it belongs to the same category as its neighbours (uniformity). We used a reference set of 2,048 scenes (64 categories), evaluated pairwise scene similarity using deep features from a pretrained convolutional neural network (CNN), and calculated the image-space sparseness and uniformity for each image. We ran three memory experiments, varying the memory workload with experiment length and colour/greyscale presentation. We measured the sensitivity and criterion value changes as a function of image-space sparseness and uniformity. Across all three experiments, we found separate effects of 1) sparseness on memory sensitivity, and 2) uniformity on the recognition criterion. People better remembered (and correctly rejected) images that were more separated from others. People tended to make more false alarms and fewer miss errors in images from categorically uniform portions of the image-space. We propose that both image-space properties affect human decisions when recognising images. Additionally, we found that colour presentation did not yield better memory performance over grayscale images.

show abstract

Dynamics of scene representations in the human brain revealed by magnetoencephalography and deep neural networks

Cited by 52 publications

References 62 publications

Exploring spatiotemporal neural dynamics of the human visual cortex

Exploring spatiotemporal neural dynamics of the human visual cortex

Cortical sensitivity to natural scene structure

Visual properties and memorising scenes: Effects of image-space sparseness and uniformity

Contact Info

Product

Resources

About