Visual features as stepping stones toward semantics: Explaining object similarity in IT and perception with non-negative least squares

Abstract

similarity, in brain representations and conscious perception, must reflect a combination of the visual appearance of the objects on the one hand and the categories the objects belong to on the other.Indeed, visual object features and category membership have each been shown to contribute to the object representation in human inferior temporal (IT) cortex, as well as to object-similarity judgments. However, the explanatory power of features and categories has not been directly compared. Here, we investigate whetherthe IT object representation and similarity judgments are best explained by a categorical or a feature-based model.We use rich models (> 100 dimensions) generated by human observers for a set of 96 real-world object images. The categorical model consists of a hierarchically nested set of category labels (such as "human", "mammal", "animal"). The feature model includes both object parts (such as "eye", "tail", "handle") and other descriptive features (such as "circular", "green", "stubbly"). We used nonnegative least squares to fit the models to the brain representations (estimated from functional magnetic resonance imaging data) and to similarity judgments. Model performance was estimated on held-out images not used in fitting. Both modelsexplained significant variance in ITand the amounts explained were not significantly different.The combined model did not explain significant additional IT variance, suggesting that it is the shared model variance (features correlated with categories, categories correlated with features) that best explains IT. The similarity judgments were almost fully explained by the categorical model, which explained significantly more variance than the feature-based model. The combined model did not explain significant additional variance in the similarity judgments. Our findings suggest that IT uses features that help to distinguish categories as stepping stones toward a semantic representation.Similarity judgments contain additional categorical variance that is not explained by visual features, reflecting a higher-level more purely semantic representation.

An image-computable model of human visual shape similarity

“…Mean, standard deviation, skew, and kurtosis of (30)(31)(32)(33)(34)(35)(36)(37)(38). Motivated by Zhang and Lu (2004…”

Analysing linear multivariate pattern transformations in neuroimaging data

“…The fMRI data set has been used in previous publications (Kriegeskorte et al, 2008a;Kriegeskorte, et al, 2008b;Mur et al, 2012;Mur et al 2013;Jozwik et al 2016). Four healthy human volunteers participated in the fMRI experiment (mean age 35 years; two females).…”

“…We applied our approach to an existing event-related fMRI data set that has been used in several previous publications to address different conceptual questions (Kriegeskorte et al 2008a;Kriegeskorte et al 2008b;Mur et al 2012;Mur et al 2013;Jozwik et al 2016). Four human participants were presented with 96 photographic images of faces (24 images), places (8 images) and objects (64 images).…”

A data‐driven approach to stimulus selection reveals an image‐based representation of objects in high‐level visual areas

“…However, the sparse nature of these recordings makes it difficult to determine with certainty the critical dimensions along which information is represented. A number of neuroimaging studies have directly compared the influence of low-level and high-level properties on patterns of response to objects (Kriegeskorte et al, 2008;Lescroat and Gallant, 2019;Naselaris et al, 2009;Clarke & Tyler, 2014;Bracci & Op de Beeck, 2016;Proklova, Kaiser, & Peelen, 2016;Jozwik, Kriegeskorte, & Mur, 2016). For example, categoryselective patterns of response are still evident when images have been scrambled in a way that preserves some of their visual properties, but removes their semantic properties (Andrews et al, 2010;Coggan, Liu, Baker, & Andrews, 2016;Long, Yu, & Konkle, 2018;Watson, Andrews, & Hartley, 2017).…”

“…For example, categoryselective patterns of response are still evident when images have been scrambled in a way that preserves some of their visual properties, but removes their semantic properties (Andrews et al, 2010;Coggan, Liu, Baker, & Andrews, 2016;Long, Yu, & Konkle, 2018;Watson, Andrews, & Hartley, 2017). A number of neuroimaging studies have directly compared the influence of low-level and high-level properties on patterns of response to objects (Kriegeskorte et al, 2008;Lescroat and Gallant, 2019;Naselaris et al, 2009;Clarke & Tyler, 2014;Bracci & Op de Beeck, 2016;Proklova, Kaiser, & Peelen, 2016;Jozwik, Kriegeskorte, & Mur, 2016). It is becoming clear from these studies that the representation across the ventral stream reflects both high-level and low-level representations.…”