Interpretation of Social Interactions: Functional Imaging of Cognitive-Semiotic Categories During Naturalistic Viewing

Abstract

Social interactions arise from patterns of communicative signs, whose perception and interpretation require a multitude of cognitive functions. The semiotic framework of Peirce’s Universal Categories (UCs) laid ground for a novel cognitive-semiotic typology of social interactions. During functional magnetic resonance imaging (fMRI), 16 volunteers watched a movie narrative encompassing verbal and non-verbal social interactions. Three types of non-verbal interactions were coded (“unresolved,” “non-habitual,” and “habitual”) based on a typology reflecting Peirce’s UCs. As expected, the auditory cortex responded to verbal interactions, but non-verbal interactions modulated temporal areas as well. Conceivably, when speech was lacking, ambiguous visual information (unresolved interactions) primed auditory processing in contrast to learned behavioral patterns (habitual interactions). The latter recruited a parahippocampal-occipital network supporting conceptual processing and associative memory retrieval. Requesting semiotic contextualization, non-habitual interactions activated visuo-spatial and contextual rule-learning areas such as the temporo-parietal junction and right lateral prefrontal cortex. In summary, the cognitive-semiotic typology reflected distinct sensory and association networks underlying the interpretation of observed non-verbal social interactions.

The Bidirectional Social-Cognitive Mechanisms of the Social-Attention Symptoms of Autism

“…Because joint attention and social cognition play an integral role in perspective taking that involves the integrated triadic processing of self-referenced and other-referenced information, they are thought to involve the processing across a widely distributed frontal, temporal, parietal system [( 20 ), cf. ( 216 )]. In addition, joint attention and social orienting are thought to involve motivation processes supported by mid-brain reward networks ( 20 ) and the interaction between bottom-up salience network regulation of attention and top-down executive control of social attention ( 79 ) that begins between 3 and 4 months of age ( 16 , 41 , 217 , 218 ).…”

Movies and narratives as naturalistic stimuli in neuroimaging

“…Providing causal evidence of the involvement of pSTS, transcranial magnetic stimulation over the pSTS reduced fMRI responses to three-second clips of moving faces in the pSTS ( Pitcher et al, 2017 ). The involvement of pSTS/TPJ in perception of social cues has been further clarified in a study that contrasted activations caused by movie clips of unresolved interactions ( e.g ., two people wordlessly staring at each other), non-habitual interactions ( e.g ., a pedestrian patting a cyclist on the shoulder at a traffic light), and habitual interactions ( e.g ., paying for groceries at a store) ( Wolf et al, 2018 ). Notably, movie clips are very well suited for contrasting fMRI activity between these different types of interactions, and it would have been difficult to study this with artificial stimuli.…”

Digital Storytelling in Early Childhood: Student Illustrations Shaping Social Interactions

“…In educational settings, different technological tools and programs, such as podcasts, infographics, and other types of presentations, make it easy for instructors to create digital stories ( McLellan, 2007 ; Bower, 2015 ). Digital stories weave “the art of telling stories with a variety of digital multimedia, such as images, audio, and video” ( Robin, 2006 , p. 1).…”