Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing

Masson, Haemy Lee; Işık, Leyla

doi:10.1016/j.neuroimage.2021.118741

Cited by 40 publications

(46 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When conducting the headwise encoding analyses (Fig. 6), we also discard the learned coefficients corresponding to all heads except for the particular head of interest for prediction and evaluation on the test set (Lee Masson & Isik, 2021).…”

Section: Encoding Model Estimation and Evaluationmentioning

confidence: 99%

Shared functional specialization in transformer-based language models and the human brain

Kumar

Sumers

Yamakoshi

et al. 2022

Preprint

View full text Add to dashboard Cite

Piecing together the meaning of a narrative requires understanding not only the individual words but also the intricate relationships between them. How does the brain construct this kind of rich, contextual meaning from natural language? Recently, a new class of artificial neural networks—based on the Transformer architecture—has revolutionized the field of language modeling. Transformers integrate information across words via multiple layers of structured circuit computations, forming increasingly contextualized representations of linguistic content. In this paper, we deconstruct these circuit computations and analyze the associated "transformations" (alongside the more commonly studied "embeddings") at each layer to provide a fine-grained window onto linguistic computations in the human brain. Using functional MRI data acquired while participants listened to naturalistic spoken stories, we find that these transformations capture a hierarchy of linguistic computations across cortex, with transformations at later layers in the model mapping onto higher-level language areas in the brain. We then decompose these transformations into individual, functionally-specialized "attention heads" and demonstrate that the emergent syntactic computations performed by individual heads correlate with predictions of brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers, contextual distances, and syntactic dependencies in a low-dimensional cortical space. Our findings provide a new basis for using the internal structure of large language models to better capture the cascade of cortical computations that support natural language comprehension.

show abstract

Section: Encoding Model Estimation and Evaluationmentioning

confidence: 99%

Shared functional specialization in transformer-based language models and the human brain

Kumar

Sumers

Yamakoshi

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The sensitivity of the SI-pSTS to both interaction and motion is in stark contrast to the response in nearby TPJ, which is neither driven by these rich social stimuli, nor sensitive to manipulations of either interactivity or motion. While some prior work has suggested that TPJ is involved in the processing of complex scenes of social interaction (e.g., Canessa et al, 2012), it seems likely that this occurs primarily when social scenes or the task require mentalising (Masson & Isik, 2021;Walbrin et al, 2018), rather than when such scenes are unambiguous and participants are performing a non-social orthogonal task, as in the current work.…”

Section: Discussionmentioning

confidence: 69%

“…Wagner et al (2016) found that medial prefrontal cortex (mPFC) was most involved in processing naturalistic social interactions and speculated that this was because social interactions trigger spontaneous mentalising processes. Masson and Isik (2021), however, found bilateral pSTS to be most involved in social interaction perception. They assessed the contribution of both mentalising processes and social interaction perception, concluding that social interactions explained unique variance in the response of the pSTS to naturalistic scenes in a way that was not replicated in "canonical" ToM regions, including mPFC.…”

Section: Introductionmentioning

confidence: 83%

“…Indeed, recent work suggests that observed interactions are perceived and processed in a way that extends beyond the sum of perceiving multiple individuals (Abassi & Papeo, 2020, 2021Bellot et al, 2021;Isik et al, 2017;Masson & Isik, 2021;Walbrin et al, 2018;Walbrin & Koldewyn, 2019). Studies focused on dynamic interactions (Isik et al, 2017;Masson & Isik, 2021;Walbrin et al, 2018) have highlighted a region of the posterior superior temporal sulcus (pSTS) as being selectively involved in the perception of observed social interactions. Notably, this region is engaged even by stimuli that contain minimal or even no realistic human visual features.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The role of motion in the neural representation of social interactions in the posterior temporal cortex

Landsiedel

Daughters

Downing

et al. 2022

Preprint

View full text Add to dashboard Cite

Humans are an inherently social species, with multiple focal brain regions sensitive to various visual social cues such as faces, bodies, and biological motion. More recently, research has begun to investigate how the brain responds to more complex, naturalistic social scenes, identifying a region in the posterior superior temporal sulcus (SI-pSTS; i.e., social interaction pSTS), among others, as an important region for processing social interaction. This research, however, has presented images or videos, and thus the contribution of motion to social interaction perception in these brain regions is not yet understood. In the current study, 22 participants viewed videos, image sequences, scrambled image sequences and static images of either social interactions or non-social independent actions. Combining univariate and multivariate analyses, we confirm that bilateral SI-pSTS plays a central role in dynamic social interaction perception but is much less involved when ‘interactiveness’ is conveyed solely with static cues. Regions in the social brain, including SI-pSTS and extrastriate body area (EBA), showed sensitivity to both motion and interactive content. While SI-pSTS is somewhat more tuned to video interactions than is EBA, both bilateral SI-pSTS and EBA showed a greater response to social interactions compared to non-interactions and both regions responded more strongly to videos than static images. Indeed, both regions showed higher responses to interactions than independent actions in videos and intact sequences, but not in other conditions. Exploratory multivariate regression analyses suggest that selectivity for simple visual motion does not in itself drive interactive sensitivity in either SI-pSTS or EBA. Rather, selectivity for interactions expressed in point-light animations, and selectivity for static images of bodies, make positive and independent contributions to this effect across the LOTC region. Our results strongly suggest that EBA and SI-pSTS work together during dynamic interaction perception, at least when interactive information is conveyed primarily via body information. As such, our results are also in line with proposals of a third visual stream supporting dynamic social scene perception.

show abstract

“…That study controlled the inter-individual consistency of the stimuli perception during the stimuli selection process, which was not done in the current study, and may explain the discrepancy. Another study (Lee Masson and Isik, 2021) examined the fMRI responses during naturalistic movie watching using encoding models, and found that a social-affective model (including features of an agent speaking, social interactions, theory of mind, perceived valence and arousal) significantly explained the fMRI response in the left STS across two different movies. This cluster fully overlapped with our L mSTS cluster.…”

Section: Discussionmentioning

confidence: 99%

Subjective understanding of actions and emotions requires interaction of the semantic and action observation networks

Zhan

Rw²,

B³

2021

Preprint

View full text Add to dashboard Cite

How we subjectively generate an understanding of other people's bodily actions and emotions is not well understood. In this 7T fMRI study, we examined the representational geometry of bodily action- and emotion-understanding by mapping individual subjective reports with word embeddings, besides using conventional univariate/multivariate analyses with predefined categories. Dimensionality reduction revealed that the representations for perceived action and emotion were high dimensional, each correlated to but were not reducible to the predefined action and emotion categories. With searchlight representational similarity analysis, we found the left middle superior temporal sulcus and left dorsal premotor cortex corresponded to the subjective action and emotion representations. Furthermore using task-residual functional connectivity and hierarchical clustering, we found that areas in the action observation network and the semantic/default-mode network were functionally connected to these two seed regions and showed similar representations. Our study provides direct evidence that both networks were concurrently involved in subjective action and emotion understanding.

show abstract

Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing

Cited by 40 publications

References 69 publications

Shared functional specialization in transformer-based language models and the human brain

Shared functional specialization in transformer-based language models and the human brain

The role of motion in the neural representation of social interactions in the posterior temporal cortex

Subjective understanding of actions and emotions requires interaction of the semantic and action observation networks

Contact Info

Product

Resources

About