Eyes on Emotion: Dynamic Gaze Allocation During Emotion Perception From Speech-Like Stimuli

Abstract

<jats:title>Abstract</jats:title> <jats:p>The majority of emotional expressions used in daily communication are multimodal and dynamic in nature. Consequently, one would expect that human observers utilize specific perceptual strategies to process emotions and to handle the multimodal and dynamic nature of emotions. However, our present knowledge on these strategies is scarce, primarily because most studies on emotion perception have not fully covered this variation, and instead used static and/or unimodal stimuli with few emotion categories. To resolve this knowledge gap, the present study examined how dynamic emotional auditory and visual information is integrated into a unified percept. Since there is a broad spectrum of possible forms of integration, both eye movements and accuracy of emotion identification were evaluated while observers performed an emotion identification task in one of three conditions: audio-only, visual-only video, or audiovisual video. In terms of adaptations of perceptual strategies, eye movement results showed a shift in fixations toward the eyes and away from the nose and mouth when audio is added. Notably, in terms of task performance, audio-only performance was mostly significantly worse than video-only and audiovisual performances, but performance in the latter two conditions was often not different. These results suggest that individuals flexibly and momentarily adapt their perceptual strategies to changes in the available information for emotion recognition, and these changes can be comprehensively quantified with eye tracking.</jats:p>

Keeping track of emotions: audiovisual integration for emotion recognition and compensation for sensory degradations captured by perceptual strategies

“…The post-hoc t-tests with Bonferroni corrected p-values showed that AV performance was higher than V (t(23) = 7.3, p < 0.001) and A (t(23) = 13.8, p < 0.001), and V was higher than A (t(23) = 9.6, p < 0.001), thus replicating our previous results 114 . Additionally, AV performance was higher than conditions with degraded visual information (AdV: t(23) = 3.8, p = 0.01; dAdV: t(23) = 4.7, p = 0.001), but not with only degraded auditory information (dAV: t(23) = 0.43, p = 1.0).…”

“…This is in line with findings in macular degeneration patients 132 and in control observers with simulated scotoma's 133,134 , and suggests that the participants in the current study developed perceptual strategies that are similar to what is seen with a preferred retinal locus (PRL) in patients. In a previous study 114 , we have shown that observers generally fixate on the face when identifying emotions. Considering the small fixation distance to the face AOIs for intact visual stimuli and the large fixation distance to the hand AOIs, it can be assumed that participants in the current study also mainly fixated on or near the face.…”

“…The stimuli and methods used in this study are directly based on and modified from previous studies by the authors and by the creators of the stimulus materials 36,114 . In the previous study by de Boer et al emotion recognition performance and gaze behavior were studied in young, healthy observers that viewed the stimuli audiovisually, only the video, or only the audio.…”

“…The reason for using many emotions was to avoid any ceiling effects that are often found in emotion research 56,122,123 , as changes in performance due to the degradations may not be entirely visible if normal performance is close to ceiling. Portrayals from two actors that were found to be less clearly recognizable in our previous work 114 were used as practice material to acquaint participants with the stimulus materials and the task. Thus, this resulted in a total of 96 unique stimuli used in the main experiment and a total of 24 unique stimuli used in practice trials.…”

“…This can be quantified by examining differences in eye movements for individuals with and without vision/hearing loss. Gaze allocation is proposed to be a functional information-seeking process 45,46,114 . Therefore, it can be expected that gaze adapts to the changes in information due to degraded visual and auditory signals.…”