Multisensory Processes: A Balancing Act across the Lifespan

Abstract

Multisensory processes are fundamental in scaffolding perception, cognition, learning and behaviour. How and when stimuli from different sensory modalities are integrated rather than treated as separate entities is poorly understood. We review how the relative reliance on stimulus characteristics versus learned associations dynamically shapes multisensory processes. We illustrate the dynamism in multisensory function across two timescales: one long-term that operates across the lifespan and one short-term that operates during the learning of new multisensory relations. In addition, we highlight the importance of task contingencies. We conclude that these highly dynamic multisensory processes, based on the relative weighting of stimulus characteristics and learned associations, provide both stability and flexibility to brain functions over a wide range of temporal scales.

Auditory object perception: A neurobiological model and prospective review

“…In other words, humans may evoke a number of different sound processing strategies relative to other animals with dependence on listening task demands, intentional or unintentional influences by language systems, and/or our ability to decode a variety of high order signal attributes that may not be behaviorally relevant to other species. Notwithstanding, many models of sensory perception acknowledge that object representations are quintessentially multisensory in global organization (Calvert and Lewis, 2004; Lewis, 2010; Murray et al., 2016) and we further assert that the auditory system in humans is also shaped by sensory-motor organizations in the brain that relate to acoustic communication. In the current review, we present a fundamental neurobiological model of hearing perception for natural sounds that addresses the question of what an “auditory object” is and the degree to which this is a useful concept for thinking about the cortical mechanisms that mediate hearing perception.…”

Behavioral and Neural Foundations of Multisensory Face-Voice Perception in Infancy

“…In adults, neural attenuation for bimodal synchrony is thought to reflect more efficient processing and has been documented in both early sensory and later attentional responses (Belse et al, 2004;Pilling, 2009;van Wassenhove, Grant, & Poepple, 2005), whereas attenuation has only been observed in later going attentional responses in infants (Hyde et al, 2011, Grossmann et al, 2006Reynolds et al, 2014;Vogel et al, 2012). Inconsistencies in the functional brain response between different experimental contexts in infants should not be taken as a limitation of the work, as most current views of multimodal processing and intersensory perception in adults conclude that context has a large effect on how, when, and where the brain is engaged (e.g., van Atteveldt et al, 2014;Matusz, Retsa, & Murray, 2016;Murray, Lewkowicz, Amedi, & Wallace, 2016). However, the fact that neural attenuation for bimodal synchrony has been documented exclusively in later attentional processing in infants but often appears during earlier sensory processing in adults may reflect a genuine developmental difference between infants and adults in efficiency and time course.…”

A multisensory perspective on object memory

“…For visual memory, the improvements ranged between 0.5% and 7% (Thelen, Matusz, & Murray, 2014), while for auditory memory these gains were between 2.5% and 10.8% (Thelen et al 2014, Supplemental Information). These findings suggest that multisensory pairings involving merely the simultaneity of stimulus onsets across the senses may only be weakly effective in supporting object memory (cf., De Meo, Murray, Clarke, & Matusz, 2015; Murray et al, 2016a,b; ten Oever et al, 2016 for reviews on the role of audiovisual simultaneity detection in modulating instantaneous perception and selective attention). We discuss these results in more detail below in the section on individual differences (Section 3).…”

“…Nonetheless and first, it is now well established that visual areas such as the LOC and auditory areas such as the STC demonstrate multisensory convergence and integration (see, e.g., Matusz, Retsa, & Murray, 2016; Sarmiento, Matusz, Sanabria, & Murray, 2015; reviewed in Doehrmann & Naumer, 2008; Murray et al, 2016b; ten Oever et al, 2016). Second, microelectrode recordings in monkey posterior infero-temporal (IT) cortex, for which the LOC is considered to be the human homologue, as well as visual area V4, demonstrate selective delay-period responses on a delayed match-to-sample task for specific multisensory and unisensory pairings (e.g., Colombo & Gross, 1994; Gibson & Maunsell, 1997; Haenny, Maunsell, & Schiller, 1988; Maunsell, Sclar, Nealey, & DePriest, 1991; see also Goulet & Murray, 2001).…”