Contextual associations represented both in neural networks and human behavior

Abstract

Contextual associations facilitate object recognition in human vision. However, the role of context in artificial vision remains elusive as does the characteristics that humans use to define context. We investigated whether contextually related objects (bicycle-helmet) are represented more similarly in convolutional neural networks (CNNs) used for image understanding than unrelated objects (bicycle-fork). Stimuli were of objects against a white background and consisted of a diverse set of contexts (N=73). CNN representations of contextually related objects were more similar to one another than to unrelated objects across all CNN layers. Critically, the similarity found in CNNs correlated with human behavior across three experiments assessing contextual relatedness, emerging significant only in the later layers. The results demonstrate that context is inherently represented in CNNs as a result of object recognition training, and that the representation in the later layers of the network tap into the contextual regularities that predict human behavior.

Fine-grained knowledge about manipulable objects is well-predicted by CLIP

“…These results extend well-established demonstrations that deep neural networks represent broad characteristics of visual object recognition (e.g. Aminoff et al, 2022;Geirhos et al, 2021;Kubilius et al, 2016;Lee & Almeida, 2021;Mukherjee & Rogers, 2023;Tuli et al, 2021;Xu & Vaziri-Pashkam et al 2021;Zeman et al, 2020;Zhou et al, 2022).…”

“…Strong performance for vision dimensions is relatively unsurprising given the vast exposure to image information during pre-training. Notably, functional information about objects is also, to some extent, accessible via visual information: This is shown to some extent with simpler pre-training schemes for CNNs that can learn contextual associations between objects (Aminoff et al, 2022;. Beyond visual information, the linguistic contributions of CLIP likely contribute to the improved approximation of functional information relative to vision-only models (see fig.…”

“…Beyond the exposure to a vast training dataset that may facilitate the learning of statistical regularities, the combination of visual and linguistic information employed by CLIP allows for the acquisition of deeper semantic representations: This likely accounts for recent demonstrations that CLIP outperforms comparison networks in the prediction of neural object responses (Contier et al, 2023;Wang et al, 2023;Zhou et al, 2022). Accordingly, CLIP is potentially well-suited to learn relatively subtle information about objects -such as manipulable object dimensions -beyond simpler semantic distinctions already shown by neural networks (Aminoff et al 2022;.…”

“…However, neural networks may still acquire some aspects of human object knowledge. Studies that probe the representational contents of pre-trained neural networks reveal evidence of "incidental learning" of information that is not explicitly intended: For instance, convolutional neural networks (CNNs) trained on an a simple image classification task can learn certain semantic associations -for example, image representations for bicycles are more similar to helmets than semantically unrelated objects, such as forks (Aminoff et al, 2022), and fish are more similar to underwater scenes than land scenes .…”

“…Modeling human-to-neural-network correspondence in this way may have several advantages over representational similarity analysis (RSA) -a common approach for human-to-neural-network comparisons (e.g. Aminoff et al, 2022;Xu & Vaziri-Pashkam et al 2021;Zeman et al, 2020). First, RSA measures the overall similarity of two measures or systems by comparing their multi-dimensional representational spaces, but does not consider the constituent dimensions that structure these spaces: The potential interpretational value of these dimensions is overlooked -for example, understanding that human-network correspondence is driven by a dimension consistent with "elongation" is more informative than broadly construed "visual properties".…”