Examining the Coding Strength of Object Identity and Nonidentity Features in Human Occipito-Temporal Cortex and Convolutional Neural Networks

Abstract

A visual object is characterized by multiple visual features, including its identity, position and size. Despite the usefulness of identity and nonidentity features in vision and their joint coding throughout the primate ventral visual processing pathway, they have so far been studied relatively independently. Here in both female and male human participants, the coding of identity and nonidentity features was examined together across the human ventral visual pathway. The nonidentity features tested included two Euclidean features (position and size) and two non-Euclidean features (image statistics and spatial frequency (SF) content of an image). Overall, identity representation increased and nonidentity feature representation decreased along the ventral visual pathway, with identity outweighing the non-Euclidean but not the Euclidean features at higher levels of visual processing. In 14 convolutional neural networks (CNNs) pretrained for object categorization with varying architecture, depth, and with/without recurrent processing, nonidentity feature representation showed an initial large increase from early to mid-stage of processing, followed by a decrease at later stages of processing, different from brain responses. Additionally, from lower to higher levels of visual processing, position became more underrepresented and image statistics and SF became more overrepresented compared with identity in CNNs than in the human brain. Similar results were obtained in a CNN trained with stylized images that emphasized shape representations. Overall, by measuring the coding strength of object identity and nonidentity features together, our approach provides a new tool for characterizing feature coding in the human brain and the correspondence between the brain and CNNs.

Mixed Evidence for Gestalt Grouping in Deep Neural Networks

“…In addition, DNNs often act in surprising non-human-like ways, such as being fooled by adversarial images (Szegedy et al, 2013;Dujmović et al, 2020) and make bizarre classification errors to familiar objects in unusual poses (Kauderer-Abrams, 2017;Gong et al, 2014;Chen et al, 2017). Furthermore, recent work by Xu and Vaziri-Pashkam (2021) failed to find strong neural correlates on high level visual areas when DNNs' internal representation were compared to fMRI data of human participants.…”

Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training

“…We believe that this is due to their reliance of Euclidean distance which underestimated the degree of scale invariance and subsequently (since the transformations were aggregated) the strength of all invariances. Also Xu and Vaziri-Pashkam (2021) found inconsistent embeddings across classes of translated objects, which we believe to be due to the different training setup they employed. We expand on this in the Supplementary Material.…”

Joint representation of color and form in convolutional neural networks: A stimulus-rich network perspective

“…The layers were also chosen in such a manner as to sample the network as evenly as possible, and to at least roughly equate the number of layers extracted from each network. In a control analysis, we found that our sampled layers capture the overall processing trajectory of the network and that the trajectory does not change with the types of layers sampled, as long as they are adjacent to each other in the processing pipeline (S1 Fig) . Although fully-connected layers (including the classification layer) differ from early layers in the network in that they do not follow a weight-sharing constraint over space, past work has found that they encode not just information about object category membership, but also information about features such as shape, position, spatial frequency, and size [19,36], making it appropriate to examine how they jointly encode the features of shape and color at the end of CNN visual processing.…”

“…A priori, one would assume that the final fully connected layer encodes object category orthogonally to color, since it is trained to output category labels. However, prior work has shown that fully connected layers encode not just information about object category membership, but also information about features such as shape, position, spatial frequency, and size [ 19 , 36 ]. The present results further show that there is both a significant amount of color representation and a greater amount of color and form interaction in the final compared to the first sampled layer, with the amount of interaction steadily increasing during the course of visual processing.…”

“…This includes understanding how different types of visual features are represented together in CNNs during the course of visual processing. Several studies have examined how individual features are encoded in CNNs, with some finding that coding for object identity-irrelevant features increases in higher CNN layers [ 2 , 19 ]. Additional approaches to understanding internal CNN representations (summarized in [ 9 , 20 ]) include synthesizing images that maximally drive individual CNN units (e.g., [ 21 ]), ablating sets of units and examining how this impairs network performance (e.g., [ 22 ]), and using principal components analysis to visualize how different features are encoded in a given layer (e.g., [ 23 ]).…”