A multisensory perspective on object memory

Abstract

Traditional studies of memory and object recognition involved objects presented within a single sensory modality (i.e., purely visual or purely auditory objects). However, in naturalistic settings, objects are often evaluated and processed in a multisensory manner. This begets the question of how object representations that combine information from the different senses are created and utilised by memory functions. Here we review research that has demonstrated that a single multisensory exposure can influence memory for both visual and auditory objects. In an old/new object discrimination task, objects that were presented initially with a task-irrelevant stimulus in another sense were better remembered compared to stimuli presented alone, most notably when the two stimuli were semantically congruent. The brain discriminates between these two types of object representations within the first 100ms post-stimulus onset, indicating early “tagging” of objects/events by the brain based on the nature of their initial presentation context. Interestingly, the specific brain networks supporting the improved object recognition vary based on a variety of factors, including the effectiveness of the initial multisensory presentation and the sense that is task-relevant. We specify the requisite conditions for multisensory contexts to improve object discrimination following single exposures, and the individual differences that exist with respect to these improvements. Our results shed light onto how memory operates on the multisensory nature of object representations as well as how the brain stores and retrieves memories of objects.

The Interplay Between Multisensory Associative Learning and IQ in Children

“…Within a typical school environment there is an abundance of facilitatory multisensory information (e.g., auditory and visual speech information, and written and spoken instruction), in addition to distracting stimuli (e.g., background noise and movement). Indeed multisensory processes have been shown to enhance the speed and accuracy of responses (Barutchu, Crewther, & Crewther, ; Barutchu et al., ; Jordan & Baker, ; Mahoney, Verghese, Dumas, Wang, & Holtzer, ; Miller, ), as well as improve learning and memory throughout the life span (Botta et al., ; Broadbent, White, Mareschal, & Kirkham, ; Brunetti, Indraccolo, Mastroberardino, Spence, & Santangelo, ; Fifer, Barutchu, Shivdasani, & Crewther, ; Heikkila & Tiippana, ; Lehmann & Murray, ; Matusz, Wallace, & Murray, ; Shams & Seitz, ; Seitz, Kim, & Shams, ). Multisensory training has also been shown to enhance unisensory learning (e.g., Alais & Cass, ; Seitz et al., ).…”

Brain and Cognitive Mechanisms of Top–Down Attentional Control in a Multisensory World: Benefits of Electrical Neuroimaging

“…ten Oever et al 2016). However, 21 other insights may not generalise as easily: multisensory integration, often involuntary and effortless, is known to enhance/alter a wide range of cognitive functions: from faster, more accurate and less variable perception through stronger distraction and interference to more robust learning and memory (Ernst & Banks, 2002;Gibson et al, 1997;Kriegstein & Giraud, 2006;Kriegstein et al, 2005;Matusz et al, 2015aMatusz et al, , 2017Murray & Wallace, 2012;Sarmiento et al, 2016;Shams & Seitz, 2008;Stein, 2012;Taylor et al, 2006;Thelen et al, 2014). The scarcity of research on efficacy and strength of multisensory processes in real-world or even lab-based settings (like created here) and the non-linearity of mechanisms governing multisensory processes render attempts at modelling "signal" or "noise" in naturalistic settings without overt unisensory vs. multisensory condition manipulations limited in their validity.…”

“…Additionally, posterior parietal cortices, known to contribute to control of attention towards task-relevant spatial locations and task-relevant object features, represent space in a multisensory fashion (Assad & Maunsell, 1995;Shulman et al, 2002;reviewed in Stein & Stanford, 2008). Lastly, there is converging evidence to suggests that even neurons at such early stages of cortical processing as "sensory-specific" V1 are involved in multisensory integration (reviewed in Murray et al 2016a), making the multisensory nature of object representations in the higher-level brain areas even more plausible (Matusz et al, 2017).…”

Auditory enhancement of illusory contour perception

“…Results of research from the past ∼20 years have demonstrated that multisensory processes are pervasive and impact perception and behaviour even at the earliest cerebral stages (reviewed in (Murray, Lewkowicz, Amedi, & Wallace, 2016)). For example, auditory information can impact the excitability of primary visual cortex, can facilitate stimulus detection, and can even implicitly facilitate memory encoding and retrieval (reviewed in (Matusz, Wallace, & Murray, 2017; Murray, Thelen, et al, 2016)). Auditory information can likewise impact visual attention processes (e.g.…”

“…(Matusz & Eimer, 2011; Spence, 2010)), visual discrimination and object recognition (e.g. (Amedi, von Kriegstein, van Atteveldt, Beauchamp, & Naumer, 2005)), as well as visual recognition memory (reviewed in (Matusz et al, 2017)). All of these (and other) examples entail auditory influences on responses to or representations of physically-present visual information.…”