Effect of impoverished information on multisensory integration in judgments of learning.

Abstract

According to the principle of inverse effectiveness (PIE), weaker responses to information in one modality (i.e., unisensory) benefit more from additional information in a second modality (i.e., multisensory; Meredith & Stein, 1986). We suggest that the PIE may also inform whether perceptual fluency affects judgments of learning (JOLs). If JOLs follow the PIE, the differences in JOLs for multisensory and unisensory items should increase as the unisensory study items become harder to perceive. That is, an influence of perceptual fluency should prompt a similar, interactive pattern across perceptual responses and JOLs. In 3 experiments, we systematically varied the signal intensity or noise in 1 modality to examine how responses might change with the inclusion of information in a second modality. In Experiment 1, written words in several font sizes were sometimes accompanied by spoken equivalents. In Experiments 2 and 3, spoken words in various background noise levels were sometimes accompanied by visual speech articulations. Consistent with the PIE, the multisensory benefits in response time and/or correct identification increased as responses to unisensory information decreased. Also, the multisensory formats received higher JOLs than the unisensory formats; however, unlike the predictions from PIE, this difference did not increase as study items became harder to perceive. Experiment 3 extended this finding to participants' explicit beliefs. In multisensory settings, JOLs may rely more on theory-than data-driven processes. We suggest that broadly defined processing fluency may always contribute to JOLs, but, regarding perceptual information, JOLs appear to track perceptual attributes rather than perceptual fluency.

Semantically Congruent Bimodal Presentation with Divided-Modality Attention Accelerates Unisensory Working Memory Retrieval

“…For example, Thelen et al (2015) reported that semantically congruent and incongruent multisensory gains for auditory recognition memory performance were significantly higher (6.35% vs. −11.15%) than those for visual recognition memory performance (2.35% vs. − 3.9%). For such special auditory memory facilitation, some multisensory memory studies have suggested that less effective auditory stimuli can trigger more multisensory benefits (Matusz et al, 2017; Tatz et al, 2020). Previous multisensory studies reported an inverse effectiveness relationship between visual and auditory signals in which poorly perceptible unisensory signals demonstrated strong multisensory enhancement if presented with another unisensory signal (Meredith & Stein, 1983; Stein, et al, 2009).…”

Cooking through perceptual disfluencies: The effects of auditory and visual distortions on predicted and actual memory performance

“…These effects can be observed in the visual modality with smaller fonts (Rhodes & Castel, 2008;Undorf & Zimdahl, 2019;Yang et al, 2018), hard-to-read cursive writing (Geller et al, 2018;Undorf et al, 2017), blurred (Rosner et al, 2015;Yue et al, 2013) or masked words , words with unprimed letters (Fiacconi et al, 2020) and partially deleted object pictures (Besken, 2016). In the auditory modality, a similar effect is found for quiet words (Frank & Kuhlmann, 2017;Rhodes & Castel, 2009), words that are inter-spliced with silences (Besken & Mullligan, 2014;Susser et al, 2013), or words whose signal-to-noise ratio are decreased (Tatz et al, 2021). Even though this effect has been shown repeatedly with single-word materials, the effects of perceptual disfluencies on naturalistic, multi-modal materials remain less clear.…”

“…Each recipe consisted of audio-visual information, in which both types of cues were intact, or cues in one or both modalities were distorted. We hypothesized that the distortion manipulation should reduce JOLs more for distorted than intact materials for both visual and auditory cues if the participants are experiencing difficulties for both modalities to the same extent (Peynircioglu & Tatz, 2019;Tatz et al, 2021;Undorf et al, 2018). Alternatively, one of the cues may become more pronounced, rendering the other cue futile (Besken, 2016;Susser & Mulligan, 2015;Tatz & Peynircioglu, 2020).…”

“…In other words, participants used both of these perceptual cues similarly and simultaneously. In another study, Tatz et al (2021) presented words either in single or multiple modalities and investigated how adding a cue in one modality could compensate for weak signals in the other modality. The results revealed that participants' memory predictions increased additively for the inclusion of another modality and augmentation in the clarity of the stimulus.…”

“…Both Peynircioglu and Tatz (2019) and Tatz et al (2021) inform us about the integration of visual and auditory materials to JOLs; however, it is essential to see if these perceptual effects generalize to more naturalistic, multimodal materials. When single word-like materials are used, participants may rely more on data-driven strategies to identify items (Blank & Davis, 2016;Sohoglu et al, 2014) and to make their JOLs.…”