Memory and learning with rapid audiovisual sequences

Abstract

We examined short-term memory for sequences of visual stimuli embedded in varying multisensory contexts. In two experiments, subjects judged the structure of the visual sequences while disregarding concurrent, but task-irrelevant auditory sequences. Stimuli were eight-item sequences in which varying luminances and frequencies were presented concurrently and rapidly (at 8 Hz). Subjects judged whether the final four items in a visual sequence identically replicated the first four items. Luminances and frequencies in each sequence were either perceptually correlated (Congruent) or were unrelated to one another (Incongruent). Experiment 1 showed that, despite encouragement to ignore the auditory stream, subjects' categorization of visual sequences was strongly influenced by the accompanying auditory sequences. Moreover, this influence tracked the similarity between a stimulus's separate audio and visual sequences, demonstrating that task-irrelevant auditory sequences underwent a considerable degree of processing. Using a variant of Hebb's repetition design, Experiment 2 compared musically trained subjects and subjects who had little or no musical training on the same task as used in Experiment 1. Test sequences included some that intermittently and randomly recurred, which produced better performance than sequences that were generated anew for each trial. The auditory component of a recurring audiovisual sequence influenced musically trained subjects more than it did other subjects. This result demonstrates that stimulus-selective, task-irrelevant learning of sequences can occur even when such learning is an incidental by-product of the task being performed.

Characterizing the roles of alpha and theta oscillations in multisensory attention

“…By showing that auditory attention involves suppression of visual information, this result projects ideas about selective attention into a multisensory context. Specifically, the result is consistent with the view from work on multisensory processing that information in one sensory modality can influence perception and memory for information from another modality (Keller & Sekuler, 2015; Teramoto, Kobayashi, & Hidaka, 2013; Hidaka et al, 2009; Sekuler, Sekuler, & Lau, 1997; Shams, Kamitani, & Shimojo, 2000; Shipley, 1964; Teramoto, Hidaka, & Sugita, 2010; McGurk & MacDonald, 1976). In other words, suppression of irrelevant information in one sensory modality can advantage the processing of information in another sensory modality.…”

“…However, when attention is directed toward one modality, it may not always possible to shut out completely inputs from a different modality. For example, Keller and Sekuler (2015) presented subjects with simultaneous visual and auditory sequences, instructing them to attend the visual sequence while ignoring the accompanying auditory sequence. Despite this instruction, and despite the performance cost associated with failure to comply, subjects were unable to completely filter out the auditory sequences.…”

Long-term implicit memory for sequential auditory patterns in humans

“…The general behavioural pattern revealed here is reminiscent of the 'noise memory' effect first shown by Agus et al (2010; see also Agus & Pressnitzer, 2013;Andrillon et al, 2015;Gold, Aizenman, Bond, & Sekuler, 2014;Keller & Sekuler, 2015;Luo, Tian, Song, Zhou, & Poeppel, 2013). In that study naïve listeners readily remembered reoccurring white-noise snippets presented amongst novel noise bursts.…”

Memory and learning for visual signals in time and space

“…These links include a suggestion that information from the two dimensions of processing converges at some site in the parietal lobe (e.g., Walsh 2003;Oliveri, Koch, & Caltagirone 2009). Such convergence might support a combination or coordination of temporal and spatial streams of information, as some psychophysical studies have shown (Goldberg et al, 2015;Keller & Sekuler, 2015). Although the tasks in those studies differed from ours, their results do suggest the possibility that by making both sources of information available at the same time, concurrent spatial and temporal presentations could enhance performance over what would be produced by either source alone.…”