Impact of voice on emotional judgment of faces: An event‐related fMRI study

Abstract

Emotional information can be conveyed by various means of communication, such as propositional content, speech intonation, facial expression, and gestures. Prior studies have demonstrated that inputs from one modality can alter perception in another modality. To evaluate the impact of emotional intonation on ratings of emotional faces, a behavioral study first was carried out. Second, functional magnetic resonance (fMRI) was used to identify brain regions that mediate crossmodal effects of emotional prosody on judgments of facial expressions. In the behavioral study, subjects rated fearful and neutral facial expressions as being more fearful when accompanied by a fearful voice as compared to the same facial expressions without concomitant auditory stimulus, whereas no such influence on rating of faces was found for happy voices. In the fMRI experiment, this shift in rating of facial expressions in presence of a fearfully spoken sentence was correlated with the hemodynamic response in the left amygdala extending into the periamygdaloid cortex, which suggests that crossmodal effects on cognitive judgments of emotional information are mediated via these neuronal structures. Furthermore, significantly stronger activations were found in the mid-portion of the right fusiform gyrus during judgment of facial expressions in presence of fearful as compared to happy intonations, indicating that enhanced processing of faces within this region can be induced by the presence of threat-related information perceived via the auditory modality. Presumably, these increased extrastriate activations correspond to enhanced alertness, whereas responses within the left amygdala modulate cognitive evaluation of emotional facial expressions.

Sensitivity of the autonomic nervous system to visual and auditory affect across social and non-social domains in Williams syndrome

“…Specifically, while there is consensus of opinion that the amygdala represents a central subcortical structure involved in affective face processing, the literature is inconsistent regarding amygdala involvement in the processing of affective prosody (Scott et al, 1997;Anderson and Phelps, 1998;Wildgruber et al, 2006). However, fMRI data illustrate the recruitment of the amygdala at least in the processing of fearful or threatening stimuli, regardless of their sensory modality (Dolan et al, 2001;Ethofer et al, 2006). The current results of ANS responsivity to auditory emotional stimuli in individuals with WS indicated decreased tonic EDA changes in response to both vocal and music stimuli in WS relative to the TD, suggestive of attenuated habituation to the stimuli; however, a similar pattern was also observed in the TD group.…”

Supramodal Representation of Emotions

“…This is particularly important for the perception of emotional cues in social interactions, in which congruency between facial expression and emotional prosody facilitates emotion recognition . Emotional prosody can alter facial emotion perception (Massaro and Egan, 1996) independent from attention and even with the explicit instruction to ignore one modality (Ethofer et al, 2006a).…”

Is two better than one? A cross-modal oddball paradigm reveals greater sensitivity of the P300 to emotional face-voice associations

“…Unimodal exploration of emotion is thus nowadays considered insufficient to comprehend the complexity of 'normal' and 'pathological' emotion processing (Ethofer et al, 2006). Variants of the classical 'unimodal' oddball paradigm (using, for instance, 'neutrality' as the frequent stimulus and different emotions such as fear, happiness and sadness as the deviant stimuli) have proved to be useful to show P3b abnormalities in clinical psychopathological populations.…”