Decoding of emotion expression in the face, body and voice reveals sensory modality specific representations

Abstract

1 0 1 1 1 2 2 ABSTRACT 1 3A central issue in affective science is whether the brain represents the emotional expressions of 1 4 faces, bodies and voices as abstract categories in which auditory and visual information converge 1 5 3 1 are recognized effortlessly and responded to spontaneously when rapid adaptive actions are 3 2 required. The specifics of the subjective experience in the natural environment determine which 3 3 affective signal dominates and triggers the adaptive behavior. Rarely are the face, the whole 3 4body and the voice equally salient. That is, the actual conditions under which we react to an 3 5 angry face may be different from those of hearing an angry voice or viewing whole body 3 6 movements. For example, we see faces from close by and therefore personal familiarity may play 3 7 a role in how we react to the angry face. This is less so for the voice or the whole body, both of 3 8 which already prompt reactions when seen or heard from a distance while information about 3 9 personal identity is not yet available or needed for action preparation. Thus, the angry body 4 0 expression viewed from a distance and the angry face expression seen from closeby may each 4 1 trigger a different reaction as behavior needs to be adapted to the concrete context. Therefore, a 4 2 representation of affective meaning that is sensitive to the spatiotemporal parameters may seem 4 3 desirable rather than an abstract system of higher order concepts as traditionally envisaged by 4 4 emotion theorists (Ekman P and D Cordaro 2011); but see (Lindquist KA et al. 2012). 5Research on the brain correlates of emotion has favored the traditional notion of abstract 4 6 neural representations of basic emotions and this has also been the dominant rationale for 4 7 multisensory research. Studies comparing how not just the face but also the voice and the whole 4 8 body convey emotions have followed this overall basic emotion perspective and asked where in 4 9 7 1 basic emotions with discrete brain correlates continues to generate controversy (Kragel PA and 7 2 KS LaBar 2016; Saarimaki H et al. 2016). Second, detailed meta-analyses of crossmodal and 7 3 multisensory studies, whether they are reviewing the findings about each separate modality or 7 4 the results of crossmodal studies (Dricu M and S Fruhholz 2016; Schirmer A and R Adolphs 7 5 2017), provide a mixed picture. Furthermore, these meta-analyses also show that a number of 7 6 methodological obstacles stand in the way of valid comparisons across studies. That is, taking 7 7 into account the role of task (incidental perception, passive perception, and explicit evaluation of 7 8 emotional expression) and the use of appropriate control stimuli reduces the number of studies 7 9 that can validly be compared. Third, findings from studies that pay attention to individual 8 0 differences and to clinical aspects reveal individual differences in sensory salience and 8 1 dominance in clinical populations, for example in autism and schizophrenia. For example, (Karle 8 2 KN et al. 20...

Limb contraction drives fear perception

“…The anatomical labelling of the resulting clusters was performed according to17 the atlas of Duvernoy(Duvernoy, 1999) for a more reliable localization. Univariate results are 18 not reported in this paper, but seeVaessen et al (2019) orFigure SR3inSupplementary 19…”

Sensory encoding of emotion conveyed by the face and visual context

“…This improvement in prediction suggests that posterior STS encodes both types of representations: those that are more specific to the visual modality, as well as those that are more abstract and tied to emotion categories. Debate continues over which set of variables are encoded in this region-whether it is specialized for processing facial movement or whether it takes modality-specific information as input and maps it to supramodal representations of emotion categories 16,47 . The results of our study support the idea that neural populations in posterior STS encode both types of representations: those associated with specific modalities and supramodal abstract emotion categorizations.…”

“…Our results additionally shed light on the role of posterior STS in face processing. Debate continues over which set of variables are encoded in this region-whether it is specialized for processing facial movement or whether it takes modality-specific information as input and maps it to supramodal representations of emotion categories 17,70 . Across two studies we found that abstract emotion features derived from both facial expressions and visual context predict unique 21 components of posterior STS activity; that is, the joint encoding model based on late layers from both EmoFAN and EmoNet predicted a greater proportion of posterior STS activity than either model alone.…”

Estudio de emociones en un proceso de ruptura cognitiva a través del reconocimiento facial

“…Los movimientos de los músculos faciales parecen acompañar a los estados afectivos, lo que implica que estas expresiones se pueden relacionar con las emociones discretas (De Gelder, 2006;Ko, 2018; ENSEÑANZA DE LAS CIENCIAS, 42-1 (2024), 23-42 Vaessen et al, 2019). En este sentido, disponemos de un acceso fiable y coherente al componente emocional a través de las expresiones faciales.…”