Neural correlates of audio‐visual object recognition: Effects of implicit spatial congruency

Abstract

Multisensory integration assists us to identify objects by providing multiple cues with respect to object category and spatial location. We used a semantic audiovisual object matching task to determine the effect of spatial congruency on response behavior and fMRI brain activation. Fifteen subjects responded in a four-alternative response paradigm, which visual quadrant contained the object best matched to the sound presented. Realistic sounds based on head-related transfer functions were presented binaurally with the simulated sound source corresponding to one of the four quadrants. Following a random sequence, the location of the sound corresponded to the quadrant containing the semantically congruent target on half the trials, whereas on other trials the sound arose from an incongruent location. We examined the effects of spatial congruency on response latencies, hit-rates and fMRI responses. Preliminary behavioral results revealed a significant effect of spatial congruency on response latency or performance for stimuli with noise added. In the fMRI experiment, spatial congruency had a significant effect on the BOLD response. A cluster in the right middle and superior temporal gyrus was more activated when the auditory sound sources were spatially congruent with the semantically matching visual stimulus. In an exploratory post-hoc analysis, in which we correlated the BOLD signal with the subjects' ability to locate the sound sources, we found a significant cluster in the left inferior frontal cortex, where the BOLD response increased with sound-source localization performance. Thus spatial congruency appears to enhance the semantic integration of audiovisual object information in these brain regions.

Multisensory Integration during Short-term Music Reading Training Enhances Both Uni- and Multisensory Cortical Processing

“…Specifically, it seems that the audiovisual training resulted in an increase of the activity in the posterior STG, a region that is known to respond to concurrent audiovisual stimulation (Butler et al, 2011;Plank, Rosengarth, Song, Ellermeier, & Greenlee, 2011;Driver & Noesselt, 2008;Barraclough, Xiao, Baker, Oram, & Perrett, 2005) and is affected by musical training (Zimmerman & Lahav, 2012). On the contrary, the auditory and visual training showed an increase in the activity of the right APC a region, which, as already noted, is related to the coordination of two or more separate cognitive operations (Ramnani & Owen, 2004).…”

Rhythmicity and cross-modal temporal cues facilitate detection

“…We cannot rule out the possibility that additional factors known to promote multisensory processing, such as spatial proximity (Plank, Rosengarth, Song, Ellermeier, & Greenlee, 2012; Wallace et al, 1996) or content congruency (Beauchamp, Lee, Argall, & Martin, 2004; Van Atteveldt, Formisano, Goebel, & Blomert, 2004) also contribute to the observed effects. For example, visual input was also found to increase detectability for low intensity auditory stimuli presented simultaneously (Lovelace, Stein, & Wallace, 2003).…”

Preference for Audiovisual Speech Congruency in Superior Temporal Cortex

“…STC is sensitive to audiovisual congruencies of various kinds: spatial (i.e., sources of audiovisual inputs; e.g., Plank, Rosengarth, Song, Ellermeier, & Greenlee, 2012), temporal (i.e., onsets of sound and lip movements; e.g., Macaluso, George, Dolan, Spence, & Driver, 2004), or identity (i.e., perceived phonemes; e.g., Nath & Beauchamp, 2012;Szycik et al, 2012). We manipulated the congruency of identity in our study, but it is conceivable that differences in temporal congruency might also have contributed to the observed effect in STC, if there are differences in temporal asynchrony between McGurk stimuli and congruent videos.…”