Long-term memory representations for audio-visual scenes

Abstract

In this study, we investigated the nature of long-term memory representations for naturalistic audio-visual scenes. Whereas previous research has shown that audio-visual scenes are recognized more accurately than their unimodal counterparts, it remains unclear whether this benefit stems from audio-visually integrated long-term memory representations or a summation of independent retrieval cues. We tested two predictions for audio-visually integrated memory representations. First, we used a modeling approach to test whether recognition performance for audio-visual scenes is more accurate than would be expected from independent retrieval cues. This analysis shows that audio-visual integration is not necessary to explain the benefit of audio-visual scenes relative to purely auditory or purely visual scenes. Second, we report a series of experiments investigating the occurrence of study-test congruency effects for unimodal and audio-visual scenes. Most importantly, visually encoded information was immune to additional auditory information presented during testing, whereas auditory encoded information was susceptible to additional visual information presented during testing. This renders a true integration of visual and auditory information in long-term memory representations unlikely. In sum, our results instead provide evidence for visual dominance in long-term memory. Whereas associative auditory information is capable of enhancing memory performance, the long-term memory representations appear to be primarily visual.

Encoding variability explains the multisensory benefit in recognition memory.

“…Therefore, it seems likely that the information that was activated by the two orienting questions would be integrated into the same memory trace. Thus, our results might be consistent with the idea that integration of information benefits later memory (but see Meyerhoff et al, 2023) but such integration does not seem to be stronger for multisensory than unisensory items.…”

Remembering cinematic sequences: Boundaries disrupt memory in fast-paced visual events.

“…As we have presented all our stimuli with audio, we cannot say much about the role of audio in memory for our stimuli. As previous research suggests, audio puts an additional load on cognitive resources and may lead to consequences such as increased edit blindness (Smith & Martin-Portugues Santacreu, 2017) as the audio uses the same resources in working memory (Meitz et al, 2020; Meyerhoff et al, 2023). Future research can investigate the role of audio in event cognition to further understand the factors that affect memory for visual events.…”

Modality Influences Perceived Film Suspense But Not Time Perception

“…The effectiveness of horror films in inducing suspense and altering psychological processes such as attention (Bezdek et al, 2015) and immersion/transportation (Bezdek & Gerrig, 2017) is widely acknowledged. Auditory information in horror films might play a crucial role in this process because the audio tracks have a powerful impact on the viewers' emotional (Boltz, 2001) and attentional state (Meitz et al, 2020), as well as on transportation (Huff et al, 2023) and memory (Meyerhoff et al, 2022;Meyerhoff & Huff, 2016). Using a prospective time reproduction paradigm (Zakay, 1993), we studied how perceived suspense levels of visual and audio-visual horror film excerpts affect viewers' time perception.…”