I can see you better if I can hear you coming: Action-consistent sounds facilitate the visual detection of human gait

Abstract

Observers are remarkably sensitive to point-light displays of human movement. The Superior Temporal Sulcus (STS) and premotor cortex are implicated in the visual perception of point-light human actions and the integration of perceptual signals across modalities. These neurophysiological findings suggest that auditory information might impact visual sensitivity to point-light displays of human actions. Previous research has demonstrated that coincident, action-consistent sounds enhance visual sensitivity to the presence of coherent point-light displays of human movement. Here we ask whether visual detection sensitivity is modulated specifically by the meaningfulness of sounds that are coincident with observed point-light actions. To test this hypothesis, two psychophysical studies were conducted wherein participants detected the presence of coherent point-light walkers in a mask under unimodal or audiovisual conditions. Participants in audiovisual conditions heard either tones or actual footfalls coincident with the seen walkers' footsteps. Detection sensitivity increased when visual displays were paired with veridical auditory cues (footfalls), but not when paired with simple tones. The footfall advantage disappeared when the visual stimuli were inverted. These results suggest that the visual system makes use of auditory cues during the visual analysis of human action when there is a meaningful match between the auditory and visual cues.

Action perception via auditory information: Agent identification and discrimination with complex movement sounds

“…Yet, perception in a complex sports situation never relies on only one sense. For instance, other studies have analyzed the relation between the auditory and visual senses (Thomas & Shiffrar, 2010;Sevdalis & Keller, 2010). This was examined by presenting point-light displays with added auditory information.…”

Meaningful sounds enhance visual sensitivity to human gait regardless of synchrony

“…This linear walking motion was subsequently rendered into point-light movies. The construction of the pointlight movies is described in extensive detail elsewhere (Thomas & Shiffrar, 2010).…”

“…For example, the sounds of female footsteps impact the perception of point-light walker gender (Van der Zwan et al, 2009), and the sounds of tap dancing enhance visual sensitivity to point-light tapping feet (Arrighi et al, 2009). Research from our own lab has shown that visual sensitivity to point-light walking is enhanced when paired with footstep sounds, but not when paired with temporally synchronous but meaningfully unrelated control sounds (Thomas & Shiffrar, 2010).…”

Human biological and nonbiological point-light movements: Creation and validation of the dataset

“…integration, motor perception, and motor learning (CalvoMerino, Ehrenberg, Leung, & Haggard, 2010;Elsner, FalckYtter, & Gredebäck, 2012;Graf et al, 2007;Hodges, Hayes, Breslin, & Williams, 2005;Petrini, Holt, & Pollick, 2010;Springer, Brandstadter, & Prinz, 2013;Stadler, Springer, Parkinson, & Prinz, 2012;Thomas & Shiffrar, 2010). Furthermore, they have been applied to investigate the brain processing underlying the visual and motor properties of actions, by means of functional magnetic resonance imaging (Beauchamp, Lee, Haxby, & Martin, 2003;Grossman et al, 2000;Saygin, Wilson, Hagler, Bates, & Sereno, 2004), electroencephalography (Hirai, Watanabe, Honda, & Kakigi, 2009;Krakowski et al, 2011;Ulloa & Pineda, 2007), and transcranial magnetic stimulation (Grossman, Battelli, & Pascual-Leone, 2005;van Kemenade, Muggleton, Walsh, & Saygin, 2012).…”