The recognition of facial expressions of emotion in deaf and hearing individuals

Abstract

During real-life interactions, facial expressions of emotion are perceived dynamically with multimodal sensory information. In the absence of auditory sensory channel inputs, it is unclear how facial expressions are recognised and internally represented by deaf individuals. Few studies have investigated facial expression recognition in deaf signers using dynamic stimuli, and none have included all six basic facial expressions of emotion (anger, disgust, fear, happiness, sadness, and surprise) with stimuli fully controlled for their low-level visual properties, leaving the question of whether or not a dynamic advantage for deaf observers exists unresolved. We hypothesised, in line with the enhancement hypothesis, that the absence of auditory sensory information might have forced the visual system to better process visual (unimodal) signals, and predicted that this greater sensitivity to visual stimuli would result in better recognition performance for dynamic compared to static stimuli, and for deaf-signers compared to hearing non-signers in the dynamic condition. To this end, we performed a series of psychophysical studies with deaf signers with early-onset severe-to-profound deafness (dB loss >70) and hearing controls to estimate their ability to recognize the six basic facial expressions of emotion. Using static, dynamic, and shuffled (randomly permuted video frames of an expression) stimuli, we found that deaf observers showed similar categorization profiles and confusions across expressions compared to hearing controls (e.g., confusing surprise with fear). In contrast to our hypothesis, we found no recognition advantage for dynamic compared to static facial expressions for deaf observers. This observation shows that the decoding of dynamic facial expression emotional signals is not superior even in the deaf expert visual system, suggesting the existence of optimal signals in static facial expressions of emotion at the apex. Deaf individuals match hearing individuals in the recognition of facial expressions of emotion.

Cognition Assessment Technologies on Deaf People

“…On the other hand, and in contrast to the above, it seems that once the sign language is consolidated there are increases in subdomains of some functions, for example, in peripheral attention processes, that underlie inhibitory control (Bosworth & Dobkins, 2002), this makes sense given that a visuospatial language such as sign language develops spatially, supported more by visual and proprioceptive processes. This is consistent with the Enhanced Hypothesis proposed by Rodger et al, (2021) and Sidera et al, (2017) which states that in the absence of hearing the visual system would have a more significant visual function. However, Dye, Hauser & Bavelier (2009) suggest that hearing deprivation is not a causal factor for presenting difficulties in cognitive processes, such as attention, but that these may be a consequence of other factors that have nothing to do with deafness per se since it is essential to take into account aspects such as the etiology of deafness, sociocultural factors, the age of acquisition of a communicative system, and even the modality in which this is consolidated, that is, whether it is a sign language or an oral language.…”

Social intelligence of students with and without hearing impairment in mainstream schools

“…Children with hearing impairment experience a possible developmental delay in facial expression recognition ability (Most & Michaelis, 2012;Wang et al, 2011). Some believe this could be related to their language ability rather than deafness (Sidera et al, 2017), and some stated that people with hearing impairment match hearing individuals in the recognition of facial expressions of emotion (Rodger et al, 2021). However, students who struggle to initiate basic social awareness skills cannot form a fine peer relationship and/or interaction with others.…”

How sign language expertise can influence the effects of face masks on non-linguistic characteristics

“…For example, the mouth gesture of a smile can be semantically aligned to nouns, adjectives, and simple verbs (Bank et al, 2011 ; Johnston et al, 2016 ), but also include ancillary emotional cues which are processed by signers perceptually. Deaf signers match hearing non-signers in the recognition of facial expression of emotion (Rodger et al, 2021 ), although for them facial expressions in the first place provide grammatical and syntactic markers. Thus, some studies argue that such gesticulations may also include ancillary emotional cues which are processed by signers perceptually.…”