Degraded visual and auditory input individually impair audiovisual emotion recognition from speech-like stimuli, but no evidence for an exacerbated effect from combined degradation

Keeping track of emotions: audiovisual integration for emotion recognition and compensation for sensory degradations captured by perceptual strategies

“…Additionally, we expected that older adults would perform as well as, or even better than young adults for positive emotions, both in intact audiovisual and unimodal conditions 123,[153][154][155] . Finally, in our previous work 138 with young normal hearing adults, we found evidence for compensation, as degraded audio did not reduce performance if it was accompanied by any video, regardless of whether the video was intact or degraded. As cognitive functioning declines with age 162 , we expected that older adults might not compensate for degraded information as well as young adults.…”

“…In the present experiment, both performance and eye-tracking data were obtained to identify accuracy of emotion recognition and gaze patterns during emotion perception with dynamic stimuli, respectively. The methods, including stimuli, procedures, and analyses used in this study closely resemble those used in previous studies by the authors 114,138 . The original -unmodified -stimulus materials were first described in 36 .…”

“…In addition to the possible confounding effect of age, studies on the effects of unimodal sensory impairments on emotion recognition give little to no insight about the possible consequences of dual sensory impairments for emotion recognition, which can occur relatively frequently, i.e., up to thirty percent, in the older population 137,[144][145][146][147] . Therefore, as a first step, in a previous study 138 , we established the individual and combined effects of audio and video degradations on emotion recognition in a healthy group of young volunteers. By means of stimulus degradations, we intended to approximate some of the purely sensory and instantaneous consequences of hearing and vision impairments in simulation, that is, the lack of sensory input only, and not long-term adaptation or cognitive changes that may occur in real sensory impairments.…”

“…Therefore, aiming for good ecological validity, in our present study we presented dynamic stimulus presentation, in the form of short movie-clips, to healthy young and older adults. These stimuli and the applied stimulus degradations were the same as in our previous study 138 . The use of these simulations creates a homogeneous fictitious "patient" group, while the use of two age groups allow disentangling the effects of (simulated) hearing and vision impairment from general ageing effects.…”

“…As a step towards studying the effects of sensory impairments on emotion recognition in individuals with dual sensory impairments, here we examined the role of stimulus degradations on recognition accuracy and perceptual strategies for emotion recognition in older adults with normal hearing and vision. Additionally, we compared these with previous findings in younger adults with normal hearing and vision 138 .…”