Visual features as stepping stones toward semantics: Explaining object similarity in IT and perception with non-negative least squares

Abstract

Object similarity, in brain representations and conscious perception, must reflect a combination of the visual appearance of the objects on the one hand and the categories the objects belong to on the other. Indeed, visual object features and category membership have each been shown to contribute to the object representation in human inferior temporal (IT) cortex, as well as to object-similarity judgments. However, the explanatory power of features and categories has not been directly compared. Here, we investigate whether the IT object representation and similarity judgments are best explained by a categorical or a feature-based model. We use rich models (>100 dimensions) generated by human observers for a set of 96 real-world object images. The categorical model consists of a hierarchically nested set of category labels (such as “human”, “mammal”, and “animal”). The feature-based model includes both object parts (such as “eye”, “tail”, and “handle”) and other descriptive features (such as “circular”, “green”, and “stubbly”). We used non-negative least squares to fit the models to the brain representations (estimated from functional magnetic resonance imaging data) and to similarity judgments. Model performance was estimated on held-out images not used in fitting. Both models explained significant variance in IT and the amounts explained were not significantly different. The combined model did not explain significant additional IT variance, suggesting that it is the shared model variance (features correlated with categories, categories correlated with features) that best explains IT. The similarity judgments were almost fully explained by the categorical model, which explained significantly more variance than the feature-based model. The combined model did not explain significant additional variance in the similarity judgments. Our findings suggest that IT uses features that help to distinguish categories as stepping stones toward a semantic representation. Similarity judgments contain additional categorical variance that is not explained by visual features, reflecting a higher-level more purely semantic representation.

Getting “fumpered”: Classifying objects by what has been done to them

“…We argue that at least some of the features that define shape space are the result of ''mid-level'' perceptual organization computations that describe relationships between multiple distant locations of the object (cf. Jozwik et al, 2016;van Assen, Barla, & Fleming, 2018). Thus, texture-like representations of statistical shape features likely play an important role in inferences related to causal history.…”

“…If they are able to group objects based on the transformations, this would suggest that they access a representational space containing the relevant telltale features for this classification. We suggest that learning to categorize familiar objects establishes a feature space that provides the basis for similarity judgments and categorization of novel objects (for related accounts, see DiCarlo & Cox, 2007;Edelman & Intrator, 1997;Jozwik, Kriegeskorte, & Mur, 2016). Thus, because the feature space has been acquired before the actual experiment, observers should be able to make correct decisions from the first experimental trial on.…”

Memory influences visual cognition across multiple functional states of interactive cortical dynamics

“…Across object-sensitive occipitotemporal cortex, in addition to the center-periphery retinotopic organization, the functional cortical architecture is widely distributed and overlapping across category-specific (e.g., face) and general (nonface) areas (Gauthier, Curran, Curby, & Collins, 2003;Gauthier, Tarr, Anderson, Skudlarski, & Gore, 1999;Kanwisher, McDermott, & Chun, 1997;Kriegeskorte, Mur, & Bandettini, 2008;Kriegeskorte, Mur, Ruff, et al, 2008;Op de Beeck, Haushofer, et al, 2008;Schwarzlose, Baker, & Kanwisher, 2005;Yovel & Kanwisher, 2004). Such distributed organization seems to depend substantially on selectivity for visual features (e.g., color, shape, texture, motion, location, size) and visual similarity among objects, which structures perceptual processing and decision-making (Cichy, Kriegeskorte, Jozwik, van den Bosch, & Charest, 2019;Grill-Spector et al, 1999;Jozwik, Kriegeskorte, & Mur, 2016;Op de Beeck, Deutsch, Vanduffel, Kanwisher, & Dicarlo, 2007Op de Beeck, Wagemans, & Vogels, 2008). For example, objects within a category tend to be more visually similar to each other than objects from another category, and activity patterns across occipitotemporal cortex show an organization such that objects within a category (e.g., shoes) or a feature (e.g., color) activate a similar set of regions that is distinct but somewhat overlapping with the set of regions activated by a different category (e.g., chairs) or feature (e.g., shape) (e.g., Cichy et al, 2019;Haxby et al, 2001).…”

Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis

“…In this case, the predicted second moment can be expressed as 950 the weighted sum of different pattern components, i.e. G = i ω i C i [22,[56][57][58], with 951 the weights being free second-level parameters. In other situations, G is a nonlinear 952 function of free model parameters: For example, G depends non-linearly on the spatial 953 tuning width in population receptive field modeling [59].…”