The representational dynamics of perceived voice emotions evolve from categories to dimensions

Giordano, Bruno L.; Whiting, Caroline; Kriegeskorte, Nikolaus; Kotz, Sonja A.; Groß, Joachim; Belin, Pascal

doi:10.1038/s41562-021-01073-0

Cited by 35 publications

(35 citation statements)

References 46 publications

(65 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the results are congruent with previous studies, proposing the prolonged activation of several processing areas simultaneously for objects (Cichy et al, 2016), emotional voices (Giordano et al, 2021) and facial expressions (Muukkonen et al, 2020). The current study extends these observations to face identity representations.…”

Section: Discussionsupporting

confidence: 92%

“…By relating multivariate similarity spaces obtained from EEG and fMRI via RSA, we were able to combine the superior temporal resolution of EEG with the precise spatial resolution of fMRI. This cutting-edge analysis method, that has only been used in a hand-full of studies so far, has brought significant advances in understanding the processing of objects (Cichy et al, 2016(Cichy et al, , 2014Cichy and Pantazis, 2017;Hebart et al, 2018), biological motion (Chang et al, 2021) as well as emotions present in voices (Giordano et al, 2021) or faces (Muukkonen et al,2020;Bayer et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Prolonged and distributed processing of facial identity in the human brain

Stecher

Muukkonen

Salmela

et al. 2021

Preprint

View full text Add to dashboard Cite

The recognition of facial identity is essential for social interactions. Despite extensive prior fMRI and EEG/MEG research on the neural representations of familiar faces, we know little about the spatio-temporal dynamics of face identity information. Therefore, we applied a novel multimodal approach, by fusioning the neuronal responses recorded in an fMRI and an EEG experiment. We analyzed the neural responses to naturally varying famous faces and traced how face identity emerges over time in different areas of the brain. We found that image invariant face identity information prevails over an extended time period (from 150 to 810 ms after stimulus onset) in the representational geometry of a broadly distributed network of parietal, temporal, and frontal areas with overlapping temporal profiles. These results challenge the current hierarchical models of face perception and suggest instead concerted and parallel activation of multiple nodes in the brain's identity coding network while processing information of familiar faces.

show abstract

Section: Discussionsupporting

confidence: 92%

Section: Introductionmentioning

confidence: 99%

Prolonged and distributed processing of facial identity in the human brain

Stecher

Muukkonen

Salmela

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Our results suggest that broad-to-specific processing structure over time 7 , 13 , 55 because facial signals of dimensional information can predict specific emotion categories, but not vice versa (but see also Giordano et al. 56 for emotion vocalization processing). Broad dimensional information could also be more robust to degradation in the communication channel 57 and serve distal communication.…”

Section: Resultssupporting

confidence: 57%

“…13 These results thus raise the question of how such multiplexed signals are processed-for example, emotion category and dimensional information could be processed separately and asynchronously, with one informing the other (i.e., serving as priors). Our results suggest that broad-to-specific processing structure over time 7,13,55 because facial signals of dimensional information can predict specific emotion categories, but not vice versa (but see also Giordano et al 56 for emotion vocalization processing). Broad dimensional information could also be more robust to degradation in the communication channel 57 and serve distal communication.…”

Section: Discussionsupporting

confidence: 56%

Facial expressions elicit multiplexed perceptions of emotion categories and dimensions

et al. 2022

View full text Add to dashboard Cite

Summary Human facial expressions are complex, multi-component signals that can communicate rich information about emotions, 1 , 2 , 3 , 4 , 5 including specific categories, such as “anger,” and broader dimensions, such as “negative valence, high arousal.” 6 , 7 , 8 An enduring question is how this complex signaling is achieved. Communication theory predicts that multi-component signals could transmit each type of emotion information—i.e., specific categories and broader dimensions—via the same or different facial signal components, with implications for elucidating the system and ontology of facial expression communication. 9 We addressed this question using a communication-systems-based method that agnostically generates facial expressions and uses the receiver’s perceptions to model the specific facial signal components that represent emotion category and dimensional information to them. 10 , 11 , 12 First, we derived the facial expressions that elicit the perception of emotion categories (i.e., the six classic emotions 13 plus 19 complex emotions 3 ) and dimensions (i.e., valence and arousal) separately, in 60 individual participants. Comparison of these facial signals showed that they share subsets of components, suggesting that specific latent signals jointly represent—i.e., multiplex—categorical and dimensional information. Further examination revealed these specific latent signals and the joint information they represent. Our results—based on white Western participants, same-ethnicity face stimuli, and commonly used English emotion terms—show that facial expressions can jointly represent specific emotion categories and broad dimensions to perceivers via multiplexed facial signal components. Our results provide insights into the ontology and system of facial expression communication and a new information-theoretic framework that can characterize its complexities.

show abstract

“…Past research in the voice domain has shown that stimuli obtained through morphing between emotions can be reliably recognised by participants (Bestelmeyer et al, 2010;Laukka, 2005). Recently it was also shown that the early brain responses to these type of stimuli reflect categorical perception, while later stages of perception reflect more dimensional perception (Giordano et al, 2021). However, no study has yet investigated the perception of human voices gradually changing in both their emotional expression and sex.…”

Section: Experiments N 3: Rating Of Voicesmentioning

confidence: 99%

EMOSEX – emotion prevails over sex during implicit judgment of faces

Korb¹,

Mikuš²,

Massaccesi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Appraisals can be influenced by cultural beliefs and stereotypes. In line with this, past research has shown that judgments about the emotional expression of a face are influenced by the face’s sex, and vice versa that judgments about the sex of a person somewhat depend on the person’s facial expression. For example, participants associate anger with male faces, and female faces with happiness or sadness. However, the strength and the bidirectionality of these effects remain debated. Moreover, the interplay of a stimulus’ emotion and sex remains mostly unknown in the auditory domain. To investigate these questions, we created a novel stimulus set of 121 avatar faces and 121 human voices (available at https://bit.ly/2JkXrpy) with matched, fine-scale changes along the emotional (happy to angry) and sexual (male to female) dimensions. In a first experiment (N=76), we found clear evidence for the mutual influence of facial emotion and sex cues on ratings, and moreover for larger implicit (task-irrelevant) effects of stimulus’ emotion than of sex. These findings were replicated and extended in two preregistered studies – one laboratory categorisation study using the same face stimuli (N=108; https://osf.io/ve9an), and one online study with vocalisations (N=72; https://osf.io/vhc9g). Overall, results show that the associations of maleness-anger and femaleness-happiness exist across sensory modalities, and suggest that emotions expressed in the face and voice cannot be entirely disregarded, even when attention is mainly focused on determining stimulus’ sex. We discuss the relevance of these findings for cognitive and neural models of face and voice processing.

show abstract

The representational dynamics of perceived voice emotions evolve from categories to dimensions

Cited by 35 publications

References 46 publications

Prolonged and distributed processing of facial identity in the human brain

Prolonged and distributed processing of facial identity in the human brain

Facial expressions elicit multiplexed perceptions of emotion categories and dimensions

EMOSEX – emotion prevails over sex during implicit judgment of faces

Contact Info

Product

Resources

About