Single-trial multisensory memories affect later auditory and visual object discrimination

Abstract

Multisensory memory traces established via single-trial exposures can impact subsequent visual object recognition. This impact appears to depend on the meaningfulness of the initial multisensory pairing, implying that multisensory exposures establish distinct object representations that are accessible during later unisensory processing. Multisensory contexts may be particularly effective in influencing auditory discrimination, given the purportedly inferior recognition memory in this sensory modality. The possibility of this generalization and the equivalence of effects when memory discrimination was being performed in the visual vs. auditory modality were at the focus of this study. First, we demonstrate that visual object discrimination is affected by the context of prior multisensory encounters, replicating and extending previous findings by controlling for the probability of multisensory contexts during initial as well as repeated object presentations. Second, we provide the first evidence that single-trial multisensory memories impact subsequent auditory object discrimination. Auditory object discrimination was enhanced when initial presentations entailed semantically congruent multisensory pairs and was impaired after semantically incongruent multisensory encounters, compared to sounds that had been encountered only in a unisensory manner. Third, the impact of single-trial multisensory memories upon unisensory object discrimination was greater when the task was performed in the auditory vs. visual modality. Fourth, there was no evidence for correlation between effects of past multisensory experiences on visual and auditory processing, suggestive of largely independent object processing mechanisms between modalities. We discuss these findings in terms of the conceptual short term memory (CSTM) model and predictive coding. Our results suggest differential recruitment and modulation of conceptual memory networks according to the sensory task at hand.

Speed on the dance floor: Auditory and visual cues for musical tempo

“…This would account for the recurrence of the tempo anchoring effect in Block 4. This interpretation, where a prior auditory stimulus affects subsequent multisensory experiences, complements the findings of Thelen, Talsma, & Murray (2015) who found prior multi-sensory experience could affect the discrimination of subsequently presented auditory objects. In both their and our research, this effect depends on congruence between the auditory and visual cues.…”

Multisensory brand search: How the meaning of sounds guides consumers’ visual attention.

“…To our knowledge, the present research is the first to demonstrate that the effects of short-term multisensory learning on subsequent object recognition (e.g., Liu et al 2012;Thelen et al 2015) extend to visual search. Further, semantic links were established in Experiment 3 between initially unrelated visual and auditory cues in the learning phase; Hearing the newly associated auditory cues then improved performance in a subsequent visual search task.…”

“…Further, semantic links were established in Experiment 3 between initially unrelated visual and auditory cues in the learning phase; Hearing the newly associated auditory cues then improved performance in a subsequent visual search task. Thus, while the learning of initially unrelated audiovisual cues does not increase subsequent unisensory recognition (Lehmann and Murray 2005;Murray 2012, 2013;Thelen et al 2015), it does seem to increase multisensory integration (i.e., visual search performance was increased when the learned sound was present in the test phase).…”

“…Visual recognition of an object is facilitated (hindered) after participants have been exposed once to the visual object and its congruent (incongruent) sound (as compared to a visual-only initial presentation); such learning effects are believed only to occur when the visual and auditory stimuli are congruent during initial exposure (Murray, Foxe, and Wylie 2005;Murray 2012, 2013;Thelen, Talsma, and Murray 2015). However, Liu, Wu, and Meng (2012) observed effects of a single exposure even for ecologically unrelated stimuli: In a study phase, participants learned ecologically unrelated audiovisual associations (e.g., four meaningless tones and pictures from the Snodgrass and Vandervart, 1980, corpus) and then completed a crossmodal priming experiment.…”

The COGs (context, object, and goals) in multisensory processing

“…a dog) are categorisedas repeated more accurately if they are initially paired with congruent (e.g. abark) stimuli in the other, irrelevant sense (Murray et al 2004(Murray et al , 2005Matusz et al 2015a;Thelen et al 2015). When the same unisensory stimuli are initially paired with simple stimuli (e.g.…”