Crossmodal association of auditory and visual material properties in infants

Abstract

The human perceptual system enables us to extract visual properties of an object’s material from auditory information. In monkeys, the neural basis underlying such multisensory association develops through experience of exposure to a material; material information could be processed in the posterior inferior temporal cortex, progressively from the high-order visual areas. In humans, however, the development of this neural representation remains poorly understood. Here, we demonstrated for the first time the presence of a mapping of the auditory material property with visual material (“Metal” and “Wood”) in the right temporal region in preverbal 4- to 8-month-old infants, using near-infrared spectroscopy (NIRS). Furthermore, we found that infants acquired the audio-visual mapping for a property of the “Metal” material later than for the “Wood” material, since infants form the visual property of “Metal” material after approximately 6 months of age. These findings indicate that multisensory processing of material information induces the activation of brain areas related to sound symbolism. Our findings also indicate that the material’s familiarity might facilitate the development of multisensory processing during the first year of life.

The Other-Race-Effect on Audiovisual Speech Integration in Infants: A NIRS Study

“…We then performed statistical analyses with mean Z-scores during the 12-16 s after stimulus onset in the left and right temporal regions (Figure 4). A planned two-tailed one sample t-test with a zero response as the baseline was conducted for each region, with reference to previous studies of fNIRS in infants (e.g., Ujiie et al, 2018b). In the own-race-face condition, the concentration of oxy-Hb in the left temporal region increased significantly during both the Match [t(16) = 2.77, p = 0.028, false discovery rate ( A further analysis was conducted to examine the cortical areas that potentially exhibit brain activity related to audiovisual speech integration.…”

“…This technique is reliable and valid for measuring brain activity in infants and is also easier to conduct in infants than fMRI. Previous studies from our research group have revealed that increased hemodynamic responses of temporal regions in infants' brain in reaction to processing faces (Otsuka et al, 2007;Kobayashi et al, 2018), color (Yang et al, 2016), and audiovisual matching of material information (Ujiie et al, 2018b). Especially, it has been shown that the cerebral hemoglobin concentrations in bilateral temporal brain regions includes brain activities in the STS area (e.g., Otsuka et al, 2007;Ujiie et al, 2018b).…”

“…We assigned 17 infants to the own-race condition, and 17 infants to other-race condition. Then, we conducted measurements of brain activity in the infants using the ETG-4000 system (Hitachi Medical Systems, Tokyo, Japan), the reliability, benefit, and variability of which were validated in our previous studies (e.g., Yang et al, 2016;Ujiie et al, 2018b).…”

Development of the multisensory perception of water in infancy

“…The same interpretation can apply to the difference in timing of acquiring multisensory perception for object materials. For example, infants acquire the sound-object association for the material properties of metal after approximately 6 months of age (Ujiie et al, 2018), which is also when they develop visual perception of metal (e.g., Yang, Kanazawa, Yamaguchi, & Motoyoshi, 2015;Yang, Otsuka, Kanazawa, Yamaguchi, & Motoyoshi, 2011).…”

“…The development of multisensory perception for a material may depend on its familiarity. Ujiie et al (2018) showed that infants acquire audiovisual associations for the material properties of wood earlier than those for metal. This was demonstrated by comparing the onset of brain mapping for each material between 4 and 8 months.…”