Neural signatures of face–voice synchrony in 5‐month‐old human infants

Abstract

Infants' unitary perception of their multisensory world, including learning from people (faces and speech), hinges on temporal synchrony. Despite its importance, relatively little work has investigated the brain processes involved in infants' perception of temporal synchrony. In two experiments, we examined event-related brain potentials (ERPs) to asynchronous and synchronous audio-visual speech in infants. Both experiments showed the early auditory P2 was greater for the synchronously presented pairings and later attentional processing (Nc) was greater for asynchronous pairings. In addition, dynamic stimuli used in Experiment 2 produced a greater early visual response (N1) to the asynchronous condition and an enhanced memory-related slow wave (PSW) later for the synchronous condition. These results suggest that, like adults, auditory-visual integration for young infants begins early during sensory processing rather than later during higher-level cognitive processing. However, unlike adults, infants' brain responses may be biased towards synchrony. Furthermore, effects of attentional and memory processing confirm interpretations of behavioral looking patterns suggesting infants find synchrony more familiar.

Behavioral and Neural Foundations of Multisensory Face-Voice Perception in Infancy

“…In 5-month olds, they observed differential attentional responses (Nc), with a greater response to faces paired with congruent emotional voices compared to faces paired with incongruent emotional voices regardless of the race of the face (African American or Caucasian). Like findings reported on above, the direction of these results is consistent with some previous work (Reynolds et al, 2014) and opposite of what others have found (Knopp, 2014;Grossmann et al, 2006;Hyde et al, 2011). In contrast, 9-month olds did not show attentional response (Nc) differences by congruency in emotion between faces and voices.…”

“…There is some evidence that the neural response of infants is greater to bimodal compared to unimodal stimulation, paralleling that observed in adults and non-human animals (e.g., Besle et al, 2004;Giard & Peronnet, 1999;Pilling, 2009;Reynolds et al, 2014;Brefczynski-Lewis et al, 2009;Wallace & Stein, 1997;Wallace, Wilkinson, & Stein 1996). Infants also appear to integrate information from different senses early during sensory processing in some contexts, as has also been documented routinely in adults (Hyde et al, 2011;Besle et al, 2009). Further, later attentional responses in adults and infants distinguish between bimodal audio-visual stimulation containing redundant (e.g., synchronous) and non-redundant (e.g., asynchronous) visual face and auditory voice information (Hyde et al, 2011, Grossmann et Intersensory Perception 26 al., 2006Reynolds et al, 2014;Vogel et al, 2012;Besle et al, 2009).…”

“…Like others studies reviewed, these results provide consistent evidence for early integration of auditory and visual information in sensory processing (see Besle et al, 2009 for a review). The observed pattern of attentional response (Nc) mirrors that observed by some (Grossmann et al, 2006;Hyde et al, 2011), but is the opposite of what others have observed (Reynolds et al, 2014). Uniquely, this study raises interesting questions about how the timing of processing information from different sense modalities may interact to elicit different perceptions in infants.…”

“…Reynolds and colleagues (2014) also observed greater attentional responses (indexed by the Nc) for the synchronous compared to asynchronous audio-visual face stimuli, the opposite direction of Nc modulation observed by others (Grossmann et al, 2006;Hyde et al, 2011). Finally, Reynolds and colleagues (2014) showed great efficiency in later memory (indexed by the PSW) for the synchronous audio-visual condition similar to the results of others (Grossmann et al, 2006;Hyde et al, 2011).…”

“…From the few studies that have been conducted, it appears that infants, like adults, can integrate redundant multisensory information early in sensory processing (Bristow et al, 2009;Hyde, Jones, Flom, & Porter, 2011;Knop, 2014). For example, Hyde and colleagues (2011) presented 5-month-olds with static images of face paired with a voice saying "hi" in perfect greater response (more negative) for the asynchronous condition compared to the synchronous condition on the later-going Nc component, known to index attentional orienting in infants (see Reynolds & Richards, 2005).…”