Selective Enhancement of Object Representations through Multisensory Integration

Abstract

Objects are the fundamental building blocks of how we create a representation of the external world. One major distinction among objects is between those that are animate versus those that are inanimate. In addition, many objects are specified by more than a single sense, yet the nature by which multisensory objects are represented by the brain remains poorly understood. Using representational similarity analysis of male and female human EEG signals, we show enhanced encoding of audiovisual objects when compared with their corresponding visual and auditory objects. Surprisingly, we discovered that the often-found processing advantages for animate objects were not evident under multisensory conditions. This was due to a greater neural enhancement of inanimate objects-which are more weakly encoded under unisensory conditions. Further analysis showed that the selective enhancement of inanimate audiovisual objects corresponded with an increase in shared representations across brain areas, suggesting that the enhancement was mediated by multisensory integration. Moreover, a distance-to-bound analysis provided critical links between neural findings and behavior. Improvements in neural decoding at the individual exemplar level for audiovisual inanimate objects predicted reaction time differences between multisensory and unisensory presentations during a Go/No-Go animate categorization task. Links between neural activity and behavioral measures were most evident at intervals of 100-200 ms and 350-500 ms after stimulus presentation, corresponding to time periods associated with sensory evidence accumulation and decision-making, respectively. Collectively, these findings provide key insights into a fundamental process the brain uses to maximize the information it captures across sensory systems to perform object recognition.

Towards understanding how we pay attention in naturalistic visual search settings

“…Compared to semantically incongruent or meaningless (arbitrary) multisensory stimuli, semantically congruent stimuli are more easily identified and remembered (e.g. Laurienti et al 2004;Murray et al, 2004;Doehrmann & Naumer 2008;Chen & Spence, 2010;Matusz et al, 2015a;Tovar et al 2020;reviewed in ten Oever et al 2016;Murray et al, 2016b;Matusz et al 2020) and also, more strongly attended (Matusz et al 2015b(Matusz et al , 2019a(Matusz et al , 2019breviewed in Soto-Faraco et al, 2019;Matusz et al 2019c). For example, Iordanenscu et al (2009) demonstrated that search for naturalistic objects is faster when accompanied by irrelevant albeit congruent sounds.…”

Both stimulus‐specific and configurational features of multiple visual stimuli shape the spatial ventriloquism effect

“…Many paradigms probe multisensory binding using paradigms involving one stimulus per modality. Whether and how results obtained using these generalize to real-life multisensory object perception remains unclear (Badde et al, 2023;Bizley, Maddox, & Lee, 2016;Duarte et al, 2023;Kirsch & Kunde, 2023;Newell et al, 2023;Tovar et al, 2020). In a step towards this direction, we asked how the presence of two visual stimuli shapes multisensory perception in the audio-visual spatial ventriloquism paradigm.…”

“…We also note that the relative contribution of individual stimuli in real‐life will depend on additional factors beyond their spatio‐temporal alignment. It is known that the relative weight among two stimuli is shaped by task‐relevance, semantic congruency, prior expectations and attention (Bizley, Jones, & Town, 2016; Delong & Noppeney, 2021; Noppeney, 2021; Talsma et al, 2010; Tovar et al, 2020; Zuanazzi & Noppeney, 2020). In particular, the role of attention is difficult to dissociate from other aspects of multisensory computation, as attention may be assigned unconsciously and automatically based on multiple attributes, including the timing of individual stimuli (Donohue et al, 2015; Talsma et al, 2010; van der Burg et al, 2008b; Zhaoping, 2023).…”

Bridging the gap between believing and memory functions

“…Just as these processes evolve extremely rapidly, decision making also can evolve rapidly and below the level of conscious awareness (Figure 1). For example, during a go/no-go task, neural activity and behavioral measures were evident at intervals of 100 to 200 ms. and 350 to 500 ms. after stimulus presentation, corresponding to the time points of sensory evidence accumulation and decision making (Tovar et al, 2020).…”