MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus

Abstract

An influential neural model of face perception suggests that the posterior superior temporal sulcus (STS) is sensitive to those aspects of faces that produce transient visual changes, including facial expression. Other researchers note that recognition of expression involves multiple sensory modalities and suggest that the STS also may respond to crossmodal facial signals that change transiently. Indeed, many studies of audiovisual (AV) speech perception show STS involvement in AV speech integration. Here we examine whether these findings extend to AV emotion. We used magnetoencephalography to measure the neural responses of participants as they viewed and heard emotionally congruent fear and minimally congruent neutral face and voice stimuli. We demonstrate significant supra-additive responses (i.e., where AV > [unimodal auditory ؉ unimodal visual]) in the posterior STS within the first 250 ms for emotionally congruent AV stimuli. These findings show a role for the STS in processing crossmodal emotive signals.audio-visual emotion ͉ emotional faces ͉ emotional voices ͉ fear ͉ gamma S ome aspects of faces, such as identity, are relatively fixed whereas other aspects, such as facial expression, can change from moment to moment. A highly influential neural model of face perception suggests that the aspects of faces that change visibly (e.g., eye gaze, lip movement, facial expression) are processed by a neural pathway leading to the posterior superior temporal sulcus (STS) (1, 2). However, both lip movement and facial expression are generally accompanied by changing auditory signals. Indeed, in daily life emotion commonly is conveyed through multiple signals involving the face, voice, and body. Although originating from different sensory modalities, these transient emotive signals are perceived and integrated within our social interactions with seemingly minimal effort. It therefore is possible that recognition of emotional expression is an intrinsically crossmodal process and that the sensitivity of the posterior STS to the changeable aspects of faces may extend to voices, as the posterior STS could serve to integrate emotional signals arising from faces and voices (3). During emotion perception, the role of the posterior STS may extend beyond the visual function of perceiving facial emotion to serve a wider purpose related to the crossmodal integration of facial and vocal signals.The STS is a prime candidate for the integration of visual and auditory emotive signals. The STS lies between primary auditory and visual cortices, and studies of audiovisual (AV) speech integration have demonstrated supra-additive responses in the STS to congruent speech stimuli (4, 5). Supra-additivity is a stringent criterion for multisensory integrative regions and is based on the known electrophysiological behavior of signal integration (6, 7). Supra-additivity occurs when the observed multisensory effect exceeds the sum of the unisensory components, i.e., when AV Ͼ [unimodal auditory (A) ϩ unimodal visual (V)]. The purpose of the pr...

Distinct iEEG activity patterns in temporal-limbic and prefrontal sites induced by emotional intentionality

“…Additionally, we speculated that significant additive band-limited power effects would be evident among iEEG electrodes within the vicinity of the fMRI activity in the TL and vLPFC. In accordance with electrophysiological studies, demonstrating the early effects of affective audiovisual binding (de Gelder et al, 1999; Hagan et al, 2009; Jessen and Kotz, 2011; Pourtois et al, 2000), we predicted that the additive effect should already be evident close to stimulus onset (a few hundreds of msec; Hagan et al, 2009). Based on evidence from iEEG recordings for rapid (e.g., Krolak-Salmon et al, 2004) and brief (e.g., Sato et al, 2011b) responses to emotionally charged stimuli within the amygdala, we expected relatively fast and transient responses within the TL channels.…”

Emotion Perception: Putting the Face in Context

“…Some work has investigated the time course of vocalization context effects, with discrepant results. Research using magnetoencephalography (MEG) demonstrates that the integration of voices and faces portraying emotion occurs within 250 ms after stimulus onset in the superior temporal sulcus (Hagan et al, 2009). Yet other data suggest that the time frame of prosody effects is much slower than 250 ms.…”

Recalibration of vocal affect by a dynamic face

“…It is known that phonetic recalibration is sub-served by a brain network that includes the superior temporal sulcus (STS, see Bonte et al 2017; Kilian-Hutten et al 2011), which is also involved in audiovisual integration of emotional affect (e.g., Ethofer et al 2006; Hagan et al 2009; Klasen et al 2011). Given that comprehension of vocal affect is driven by bilateral mechanisms that involve a myriad of sensory, cognitive, and emotional processing systems (Schirmer and Kotz 2006), it may well be that STS also has a functional role in recalibration of emotional valence.…”