How the human brain exchanges information across sensory modalities to recognize other people

Abstract

Recognizing the identity of other individuals across different sensory modalities is critical for successful social interaction. In the human brain, face- and voice-sensitive areas are separate, but structurally connected. What kind of information is exchanged between these specialized areas during cross-modal recognition of other individuals is currently unclear. For faces, specific areas are sensitive to identity and to physical properties. It is an open question whether voices activate representations of face identity or physical facial properties in these areas. To address this question, we used functional magnetic resonance imaging in humans and a voice-face priming design. In this design, familiar voices were followed by morphed faces that matched or mismatched with respect to identity or physical properties. The results showed that responses in face-sensitive regions were modulated when face identity or physical properties did not match to the preceding voice. The strength of this mismatch signal depended on the level of certainty the participant had about the voice identity. This suggests that both identity and physical property information was provided by the voice to face areas. The activity and connectivity profiles differed between face-sensitive areas: (i) the occipital face area seemed to receive information about both physical properties and identity, (ii) the fusiform face area seemed to receive identity, and (iii) the anterior temporal lobe seemed to receive predominantly identity information from the voice. We interpret these results within a prediction coding scheme in which both identity and physical property information is used across sensory modalities to recognize individuals.

Activation in the angular gyrus and in the pSTS is modulated by face primes during voice recognition

“…r Effects of Face Primes on Voice Recognition r r 2559 r Our behavioral results replicate findings from two earlier studies using similar designs [Ellis et al, 1997;F€ ocker et al, 2011]. Behavioral priming effects of voice primes on face targets [Blank et al, 2015;Ellis et al, 1997] and face identity aftereffects caused by voice adaptors [Hills et al, 2010] have previously been reported, indicating bidirectional multisensory interactions. Crossmodal behavioral priming effects were further not restricted to the domain of person identity processing as they have been shown during the processing of audiovisual affective person information as well [Skuk and Schweinberger, 2013;Watson et al, 2014b].…”

“…Another alternative are priming paradigms: It is a wellestablished finding that the repeated presentation of the same stimulus (or of the same stimulus attribute) causes the fMRI signal to decline in brain regions that process that stimulus or that stimulus attribute [Grill-Spector et al, 2006;Henson, 2003;Schacter and Buckner, 1998]. Priming effects for crossmodal prime-target combinations have previously been explored with fMRI [Adam and Noppeney, 2010;Blank et al, 2015;Noppeney et al, 2008;Tal and Amedi, 2009;Watson et al, 2014b], but the effects of face primes on voice recognition have not yet been investigated.…”

Dynamic neural architecture for social knowledge retrieval

“…Research in nonhuman primates has shown that cells in anterior-ventral temporal cortex are highly sensitive to particular facial identities as well as to facial familiarity (56,57). Previous studies in humans using intracranial recording (16) or fMRI analyses (15,42,45) have suggested that the ATL can distinguish between different people using their faces, voices, or names. Our study extends these findings by using sophisticated multivariate analyses and a wider range of stimulus categories.…”

“…Next, we asked whether the ATL acts as a neural switchboard, performing in concert with other brain regions to enable the retrieval of different facets of person knowledge in a flexible and context-appropriate manner (study 2). We focus on the ATL because multiple lines of evidence from neuropsychology, electrophysiology, and neuroimaging have documented the critical role of the ATL in person identification (4,5,(11)(12)(13)(14)(15)(16), person-related learning (10,(17)(18)(19)(20)(21), semantic memory (6)(7)(8), and abstract social knowledge (1,(22)(23)(24)(25)(26)(27)(28)(29)(30)(31)(32)(33). Individuals with ATL damage due to resection or stroke have multimodal person recognition deficits (34), lose access to stored knowledge about familiar people (35,36), and have difficulties learning information about new people (4,22,37,38).…”

Cross-modal processing of voices and faces in developmental prosopagnosia and developmental phonagnosia

“…Importantly responses in the FFA appear to support the perceptual processing of voices (Blank, Kiebel, & von Kriegstein, 2015;Schall et al, 2013), providing evidence that the face and voice interact to support identity processing at earlier stages of processing than previously assumed (Bruce & Young, 1986;Burton, Bruce, & Johnston, 1990;Ellis, Jones, & Mosdell, 1997). More traditional models of person recognition propose that the face and voice undergo extensive unisensory processing and only interact to support recognition at supramodal, i.e., post-perceptual, stages of processing (Burton et al, 1990;Bruce & Young, 1986;see Blank, Wieland, and von Kriegstein, 2014;Barton and Corrow, 2016;Quaranta et al, 2016, for more recent reviews of these models).…”