Multisensory emotion perception in congenitally, early, and late deaf CI users

Abstract

Emotions are commonly recognized by combining auditory and visual signals (i.e., vocal and facial expressions). Yet it is unknown whether the ability to link emotional signals across modalities depends on early experience with audio-visual stimuli. In the present study, we investigated the role of auditory experience at different stages of development for auditory, visual, and multisensory emotion recognition abilities in three groups of adolescent and adult cochlear implant (CI) users. CI users had a different deafness onset and were compared to three groups of age- and gender-matched hearing control participants. We hypothesized that congenitally deaf (CD) but not early deaf (ED) and late deaf (LD) CI users would show reduced multisensory interactions and a higher visual dominance in emotion perception than their hearing controls. The CD (n = 7), ED (deafness onset: <3 years of age; n = 7), and LD (deafness onset: >3 years; n = 13) CI users and the control participants performed an emotion recognition task with auditory, visual, and audio-visual emotionally congruent and incongruent nonsense speech stimuli. In different blocks, participants judged either the vocal (Voice task) or the facial expressions (Face task). In the Voice task, all three CI groups performed overall less efficiently than their respective controls and experienced higher interference from incongruent facial information. Furthermore, the ED CI users benefitted more than their controls from congruent faces and the CD CI users showed an analogous trend. In the Face task, recognition efficiency of the CI users and controls did not differ. Our results suggest that CI users acquire multisensory interactions to some degree, even after congenital deafness. When judging affective prosody they appear impaired and more strongly biased by concurrent facial information than typically hearing individuals. We speculate that limitations inherent to the CI contribute to these group differences.

Multimodal and Spectral Degradation Effects on Speech and Emotion Recognition in Adult Listeners

“…The current findings are consistent with those reported in a recent investigation examining adolescent and adult CI users’ closed set emotion recognition from multisensory talker information ( Fengler et al., 2017 ). While CI users were less accurate than hearing controls in recognizing emotions from prosodic vocal cues alone, they relied more strongly on visual facial expressions as indicated by a cost to performance when visual facial cues were incongruent with the vocal expressive cues.…”

“…Our findings are also consistent with those observed by Fengler et al. (2017) in that the ceiling performance seen across spectral band conditions is indicative that multisensory emotion perception is dominated by the visual modality in actual CI users.…”

“…By contrast, when congruent sentence content and prosodic information are matched as employed in this study, a condition is created such that reliable emotion cues are distributed across both auditory and visual modalities thereby requiring the listener to attend to both modalities. Thus, studies that employ neutral sentences ( Tinnemore et al., 2018 ), semantically anomalous phrases (e.g., Dorman et al., 2016 ), or pseudowords (e.g., Fengler et al., 2017 ) in order to disentangle the segmental and prosodic contributions to emotion perception may underestimate the actual emotion perception abilities of CI users in everyday contexts. Paradigms employing ecologically valid linguistic stimuli, such as familiar and meaningful sentences, have shown a pattern of sensory-semantic integration that is distinct from that occurring in experimentally derived stimuli that do not have a meaningful linguistic context.…”

Supra-normal skills in processing of visuo-auditory prosodic information by cochlear-implanted deaf patients

“…Of interest, while CI patients present better lip-reading abilities than hearing controls ( Calvert et al, 1997 ;Rouger et al, 2007 ;Strelnikov et al, 2009 ), they are surprisingly not better, and even worse at discriminating visual prosodic cues. A similar trend has been reported in CI patients' visual discrimination ability for visual emotional prosodic information ( Fengler et al, 2017 ). To our knowledge, there was no evidence of a lower proficiency of CI patients in any visual processing.…”

“…Thus, CI patients may have developed specific skills to combine visual and auditory prosodic cues as they participate directly in speech content comprehension. However, it remains unclear why the discrimination of the emotional content in a sentence is improved in multimodal conditions similarly in CI patients and hearing participants ( Fengler et al, 2017 ) as it relies on similar acoustic changes.…”