Multidimensional processing of dynamic sounds: more than meets the ear

Abstract

Strong cross-modal interactions exist between visual and auditory processing. The relative contributions of perceptual versus decision-related processes to such interactions are only beginning to be understood. We used methodological and statistical approaches to control for potential decision-related contributions such as response interference, decisional criterion shift, and strategy selection. Participants were presented with rising-, falling-, and constant-amplitude sounds and were asked to detect change (increase or decrease) in sound amplitude while ignoring an irrelevant visual cue of a disk that grew, shrank, or stayed constant in size. Across two experiments, testing context was manipulated by varying the grouping of visual cues during testing, and cross-modal congruency showed independent perceptual and decision-related effects. Whereas a change in testing context greatly affected criterion shifts, cross-modal effects on perceptual sensitivity remained relatively consistent. In general, participants were more sensitive to increases in sound amplitude and less sensitive to sounds paired with dynamic visual cues. As compared with incongruent visual cues, congruent cues enhanced detection of amplitude decreases, but not increases. These findings suggest that the relative contributions of perceptual and decisional processing and the impacts of these processes on cross-modal interactions can vary significantly depending on asymmetries in within-modal processing, as well as consistencies in cross-modal dynamics.

Electrophysiological correlates and psychoacoustic characteristics of hearing-motion synaesthesia

“…for changes in elevation, rather the level of elevation per se). Similarly, judgments of synaesthetic loudness were not related to any properties of the visual stimuli tested here despite evidence elsewhere of, for instance, size-loudness correspondences (Liu et al, 2011).…”

“…to judge larger or looming objects as louder (Liu, Mercado, & Church, 2011); and for the presence of a visual movement to increase loudness but not pitch perception (Maniglia, Grassi, & Ward, in press). We also report the electrophysiological correlates (EEG event-related potentials) of hearing-motion synaesthesia with the aim of understanding at what stage in visual processing of the inducer group differences emerge (perceptual or post-perceptual).…”

Signal Detection Measures Cannot Distinguish Perceptual Biases from Response Biases

“…For example, the stream/bounce effect for which two objects moving towards each other are more likely to be perceived as bouncing than as streaming if a sound is presented at the moment the objects meet (Sekuler, Sekuler, & Lau, 1997) has recently been attributable to decisional processes because an effect was observed in c and not in d′ (Grove, Ashton, Kawachi, & Sakurai, 2012; see below for other examples). Even in studies that report an effect on d′, the idea that signal detection parameters can, in and of themselves, distinguish between a perceptual effect and an effect on decision processes is commonly expressed (Cardoso-Leite, Mamassian, Schutz-Bosbach, & Waszak, 2010;Lippert, Logothetis, & Kayser, 2007;Liu, Mercado, & Church, 2011;McDonald, Teder-Salejarvi, & Hillyard, 2000;Meteyard, Bahrami, & Vigliocco, 2007;Shams & Kim, 2010). This misunderstanding of the claims that can be made based on signal detection measures has led to many potentially incorrect conclusions, and it is conceivable that some stimulating and intriguing findings might not have been submitted or accepted for publication due to the misinterpretation that effects on the criterion necessarily reflected decision-based processes and not perceptual processes.…”

Familiarity with Speech Affects Cortical Processing of Auditory Distance Cues and Increases Acuity

“…Another possible factor, not considered in previous work, is that natural speech sounds are not only more familiar, they are also more reproducible. If reproducible sounds activate motor representations relevant to producing those sounds, then the availability of multimodal stimulus representations could enhance processing of acoustic cues [20], [21]. This explanation predicts that the more easily imitated a sound is, the better individuals should be able to judge how far it has traveled.…”