Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior

Abstract

Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid- and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.

Dynamic representations of faces in the human ventral visual stream link visual features to behaviour

“…Ventral stream areas encoded sustained and extensive behavioural representations as early as 120 ms after stimulus onset (Figure 7), suggesting that the extraction of features essential in behavioural decision-making is a rapid process accomplished in face-responsive cortex. This is in line with evidence found in higher-level object and scene perception (Walther et al, 2009; Bankson, Hebart, Groen, & Baker, 2018; Groen et al, 2018) and with previous studies showing that the perceptual similarity of faces is represented in neural patterns (Said, Moore, Engell, & Haxby, 2018; Furl et al, 2017).…”

“…To gain more insight into the relationship between behavioural responses, expression categories and face configuration models, we used a variance partitioning approach (Greene, Baldassano, Esteva, Beck, & Fei-fei, 2016; Groen et al, 2018). For each stimulus duration condition, the corresponding behavioural RDM was entered into a hierarchical multiple linear regression analysis, with three model RDMs as predictors: the two facial configuration models and the most correlated high-level expression model (10 ms: neutral-vs-others; 30 and 150 ms: angry-vs-others).…”

Memorability of photographs in subjective cognitive decline and mild cognitive impairment: Implications for cognitive assessment

“…To assess image similarity, we used an object classification CNN called AlexNet CNN [20]. This classification CNN is often used as a model for the human visual system, showing similarities to the brain for visual processing of objects [21] and scenes [22]. This CNN can thus approximate the neural representations of an image at different levels of extraction (i.e., low‐, mid‐, and high‐level visual features).…”

Untangling the animacy organization of occipitotemporal cortex

“…To address this, we carried out commonality analysis on the group-averaged neural RDMs for LOTC-all and VOTC-all. Commonality analysis is a method for determining how much of the explained variance of a linear model containing multiple predictors is unique to each predictor or is shared (Newton and Spurell, 1967;Siebold, McPhee, 1979) and can be used in conjunction with RSA (Groen et al 2018;Hebart et al 2018;Lescroart et al 2015). Briefly, a vector of the coefficients of determination (R 2 ) for all possible regression models (containing one, two, or three of the models as predictor)…”