Hierarchical organization of social action features along the lateral visual pathway

Abstract

Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. Here we used a data-rich approach to test the hypothesis that social action information is extracted hierarchically along this pathway. Social scenes pose several computational challenges beyond those of object recognition: they are often highly dynamic and involve relations between two or more people. To address these challenges, we curated a naturalistic stimulus set of 250 three-second videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene & object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal area (MT), mid-level visual-social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in anterior regions of the STS, even when accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content—consistent with hierarchical organization—along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the STS and lateral pathway.

Explorations of using a convolutional neural network to understand brain activations during movie watching

“…The reverse analysis further validated that feature activations linked to the lateral occipital complex were typically situated around the main characters or objects in a scene (Figure 6B). These findings align with a recent study demonstrating that responses from the lateral occipital complex were significantly predictive for scene, object, and action recognition in videos during movie viewing (McMahon et al, 2023).…”

“…However, the field still lacks machine learning models that can effectively describe various aspects of social functions due to the complexity of the naturalistic stimuli. A recent study by McMahon and colleagues employed multiple machine learning models to extract different levels of features from videos containing social interactions (McMahon et al, 2023). They established a hierarchy of social interactions, primarily linked to the temporal lobe regions.…”

Brain Dissection: fMRI-trained Networks Reveal Spatial Selectivity in the Processing of Natural Images

“…In terms of attributes, Parietal areas show a preference for colors and materials, whereas Lateral areas favor animate attributes involving people, such as 'playing,' 'standing,' and 'happy'. These preferences are largely aligned with prior results showing Parietal preferentially processes spatial visual information [22], Lateral preferentially processes social visual information [24], and Ventral preferentially processes objects and faces [23].…”

“…In particular, the human visual system is believed to be organized into distinctive processing streams: the parietal, lateral, and ventral visual streams. While previous experimental and modeling work has made progress in characterizing each of these streams [22,23,24,1,25,26], at a fine-grained level, their distinct roles in visual processing remain under-specified. Functional networks within these streams likewise remain functionally under-specified.…”

Category-selective representation of relationships in visual cortex

“…Results showed no significant correlation between the activity-based and synthetic RDM after removing low-level visual information, in all the tested ROIs except the EBA (EBA: t( 41 ). This means that in higher-level ROIs the facing/non-facing distinction was built -almost exclusively-on the visual properties of the stimuli, in line with the view that information in the physical structure of the stimuli is already relevant to the representation of social interaction (see also McMahon et al, 2023). The fact that the EBA encodes the facing/non-facing distinction beyond the low-level visual properties of the stimuli, suggests functional differences between the ROIs, with EBA serving a distinctive role in the early transformative processes of related bodies (see 2022;Gandolfo et al, 2023;Walbrin & Koldewin, 2019).…”