Sensory encoding of emotion conveyed by the face and visual context

Abstract

Humans rapidly detect and interpret sensory signals that have emotional meaning. Facial expressions—particularly important signifiers of emotion—are processed by a network of brain regions including amygdala and posterior superior temporal sulcus (pSTS). However, the precise computations these regions perform, and whether emotion-specific representations explain their responses to socially complex, dynamic stimuli remain contentious. Here we investigated whether representations from artificial neural networks (ANNs) optimized to recognize emotion from facial expressions alone or the broader visual context differ in their ability to predict human pSTS and amygdala activity. We found that representations of facial expressions were encoded in pSTS, but not the amygdala, whereas representations related to visual context were encoded in both regions. These findings demonstrate how the pSTS may operate on abstract representations of facial expressions, such as ‘fear’ and ‘joy’, whereas the amygdala encodes the emotional significance of visual information more broadly.

Understanding human amygdala function with artificial neural networks

“…Functional magnetic resonance imaging (fMRI) data for this study were sampled from the Naturalistic Neuroimaging Database (NNDb) (Aliko et al, 2020). Detailed descriptions of the participants, the paradigm used for data acquisition, and the preprocessing of the fMRI data have been described elsewhere (Aliko et al, 2020;Soderberg et al, 2023). Briefly, blood oxygen level dependent (BOLD) data from 20 subjects viewing a full-length motion picture film 500 Days of Summer was previously collected in a 1.5 T Siemens MAGNETOM Avanto with a 32 channel head coil (Siemens Healthcare, Erlangen, Germany) and consequently used for this study.…”