Object representations in the human brain reflect the co-occurrence statistics of vision and language

Abstract

A central regularity of visual perception is the co-occurrence of objects in the natural environment. Here we use machine learning and fMRI to test the hypothesis that object cooccurrence statistics are encoded in the human visual system and elicited by the perception of individual objects. We identified low-dimensional representations that capture the latent statistical structure of object co-occurrence in real-world scenes, and we mapped these statistical representations onto voxelwise fMRI responses during object viewing. We found that cortical responses to single objects were predicted by the statistical ensembles in which they typically occur, and that this link between objects and their visual contexts was made most strongly in the anterior portion of the scene-selective parahippocampal place area. In contrast, a languagebased statistical model of the co-occurrence of object names in written text predicted responses in neighboring regions of object-selective visual cortex. Together, these findings show that the sensory coding of objects in the human brain reflects the latent statistics of object context in visual and linguistic experience.

Emergent selectivity for scenes, object properties, and contour statistics in feedforward models of scene-preferring cortex

“…Our model is also not intended to explain the effects of navigational experience on the PPA, which shows stronger responses to objects that occur at navigationally important locations and treats stimuli as more similar if they come from the same place (14, 15, 48). Furthermore, our findings do not account for functional differences along the anterior-posterior extent of the PPA, which appear to reflect a general trend toward visual-form representations in the posterior PPA and mnemonic representations in the anterior PPA (31, 49, 50). However, future studies could leverage our mid-level modeling framework to test whether the effects of navigational experience in the PPA involve the modulation of mid-level features from navigationally important stimuli and to test whether the mnemonic representations of anterior PPA are implemented through the associative coding of mid-level features from co-occurring stimuli, including object categories in scenes and distinct views of places (14, 31).…”

“…We next performed a stringent test of how well our encoding models could generalize to new data by examining their ability to predict the average responses of the PPA in a separate fMRI dataset with different subjects and novel stimuli. We examined data from an fMRI study of object representation in which four subjects viewed images of isolated objects from 81 different categories (31). We found that our encoding models trained on the BOLD5000 dataset were highly accurate at predicting the fMRI responses of the PPA to novel stimuli in a completely different set of subjects (Fig.…”

“…In the scanner, the subjects performed a simple oddball-detection task of pressing a button whenever a warped object was shown. See the original publication for a detailed description of these data (31). Each object category in this dataset contained 10 unique images, which were shown in a block design.…”

“…B) A strong test of generalization performance was conducted by using the encoding models trained on BOLD5000 to predict the univariate activations of ROIs in a separate fMRI dataset with novel stimuli and different subjects. These analyses were performed using data from (31). Significant correlations between the predicted and actual fMRI responses were observed for all ROIs, showing that the encoding models trained on the BOLD5000 dataset exhibit strong generalization performance across both subjects and stimuli.…”

The human brain uses spatial schemas to represent segmented environments

“…Previous work suggests that PPA represents scenes and landmarks [71][72][73][74] but also represents associations between objects driven by spatial and temporal contiguities. [75][76][77] A recent study identified a grouping effect in right PPA and the hippocampus for objects in different rooms using a passive viewing task similar to the one we use here, 78 and other studies have identified grouping effects in hippocampus for items that are experienced together in time. 35 We suspect that grouping effects would have been stronger in the current study if we had not controlled for visual co-occurrence across segments by making all objects co-visible and controlled for temporal proximity by constraining the order in which objects were searched for during learning.…”