Preattentive processing of audio-visual emotional signals

Abstract

Previous research has shown that redundant information in faces and voices leads to faster emotional categorization compared to incongruent emotional information even when attending to only one modality. The aim of the present study was to test whether these crossmodal effects are predominantly due to a response conflict rather than interference at earlier, e.g. perceptual processing stages. In Experiment 1, participants had to categorize the valence and rate the intensity of happy, sad, angry and neutral unimodal or bimodal face-voice stimuli. They were asked to rate either the facial or vocal expression and ignore the emotion expressed in the other modality. Participants responded faster and more precisely to emotionally congruent compared to incongruent face-voice pairs in both the Attend Face and in the Attend Voice condition. Moreover, when attending to faces, emotionally congruent bimodal stimuli were more efficiently processed than unimodal visual stimuli. To study the role of a possible response conflict, Experiment 2 used a modified paradigm in which emotional and response conflicts were disentangled. Incongruency effects were significant even in the absence of response conflicts. The results suggest that emotional signals available through different sensory channels are automatically combined prior to response selection.

Multisensory emotion perception in congenitally, early, and late deaf CI users

“…The task was adapted from [5] where details of the stimulus generation and evaluation are described. Auditory stimuli consisted of short sound tracks of voices speaking out bisyllabic German pseudowords (“lolo”, “tete”, or “gigi”) at a sound level varying between 65 and 72 dB, presented via two loudspeakers located at either side of a computer screen (width = 36 cm).…”

“…For both congruent and incongruent audio-visual trials the audio track and the video stream originated from independent recordings. This allowed compensating for possible minimal temporal misalignments when independent audio and visual streams were combined ([5], for details).…”

“…In particular, redundant/congruent emotional information coming from faces and voices has been found to facilitate emotion recognition, while non-redundant/incongruent emotional information (e.g., a “happy” face and a “sad” voice) impairs judgments about the emotion expressed in one modality [1,3,4], even when response biases were controlled [5]. The finding that individuals cannot inhibit the processing of emotional input of a task-irrelevant modality has been taken as evidence for automatic interactions of crossmodal emotional signals (e.g., [4–6]).…”

“…Incongruent crossmodal stimuli were included as well since the automaticity of multisensory interactions can only be assessed with this condition (see [5], analogously to studies on audio-visual speech perception). In different blocks, participants judged the emotional expression of either the voices (i.e., affective prosodic recognition) or the faces (i.e., affective facial expression recognition).…”

Recalibration of vocal affect by a dynamic face

“…For example, a ‘fearful’ voice is more likely to be perceived as ‘fearful’ if accompanied by a ‘fearful’ face rather than a ‘happy’ one, and emotionally congruent audiovisual stimuli are responded to faster than incongruent ones (e.g., Dolan et al 2001). This type of integration happens automatically (e.g., Föcker et al 2011; Vroomen et al 2001) with quick neural consequences (e.g., Pourtois et al 2000). …”