Meaningful auditory information enhances perception of visual biological motion

Arrighi, Roberto; Marini, Francesco; Burr, David

doi:10.1167/9.4.25

Cited by 41 publications

(46 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sounds have even been shown to induce motion perception for static displays (Teramoto et al, 2010). As in our study, the meaning of the auditory signals does matter; for instance, in tap dancing, auditory interacts with visual information in a matter that depends on how synchronized the information sources are (Arrighi, Marini, & Burr, 2009). Taken together, these results suggest a plethora of interactions between the auditory system and the perception of visual motion.…”

Section: Discussionsupporting

confidence: 70%

Music can elicit a visual motion aftereffect

Hedger

Nusbaum

Lescop

et al. 2013

Atten Percept Psychophys

View full text Add to dashboard Cite

Motion aftereffects (MAEs) are thought to result from the adaptation of both subcortical and cortical systems involved in the processing of visual motion. Recently, it has been reported that the implied motion of static images in combination with linguistic descriptions of motion is sufficient to elicit an MAE, although neither factor alone is thought to directly activate visual motion areas in the brain. Given that the monotonic change of musical pitch is widely recognized in music as a metaphor for vertical motion, we investigated whether prolonged exposure to ascending or descending musical scales can also produce a visual motion aftereffect. After listening to ascending or descending musical scales, participants made decisions about the direction of visual motion in random-dot kinematogram stimuli. Metaphoric motion in the musical stimuli did affect the visual direction judgments, in that repeated exposure to rising or falling musical scales shifted participants' sensitivity to visual motion in the opposite direction. The finding that music can induce an MAE suggests that the subjective interpretation of monotonic pitch change as motion may have a perceptual foundation.

show abstract

Section: Discussionsupporting

confidence: 70%

Music can elicit a visual motion aftereffect

Hedger

Nusbaum

Lescop

et al. 2013

Atten Percept Psychophys

View full text Add to dashboard Cite

show abstract

“…Maximal enhancement occurs when visual stimuli are near the detection/discrimination threshold (28), the auditory and visual stimuli overlap in time and in space (13,29,30) and sounds carry biologically important information (27,29,31). We show that value can modulate audiovisual interactions (and, we hypothesize, multisensory interactions more generally), an effect congruent with ideas on the Bayesian brain (32)(33)(34), in which a core idea is that the brain embodies previous beliefs about the structure (i.e., statistics) of its environment.…”

Section: Discussionmentioning

confidence: 99%

Cross-modal effects of value on perceptual acuity and stimulus encoding

Pooresmaeili

FitzGerald

Bach

et al. 2014

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Cross-modal interactions are very common in perception. An important feature of many perceptual stimuli is their reward-predicting properties, the utilization of which is essential for adaptive behavior. What is unknown is whether reward associations in one sensory modality influence perception of stimuli in another modality. Here we show that auditory stimuli with high-reward associations increase the sensitivity of visual perception, even when sounds and reward associations are both irrelevant for the visual task. This increased sensitivity correlates with a change in stimulus representation in the visual cortex, indexed by increased multivariate decoding accuracy in simultaneously acquired functional MRI data. Univariate analysis showed that reward associations modulated responses in regions associated with multisensory processing in which the strength of modulation was a better predictor of the magnitude of the behavioral effect than the modulation in classical reward regions. Our findings demonstrate a value-driven crossmodal interaction that affects perception and stimulus encoding, with a resemblance to well-described modulatory effects of attention. We suggest that multisensory processing areas may mediate the transfer of value signals across senses.reward value | sensory discrimination | audiovisual T he world is structured such that objects or events that cause sensations in one sensory modality influence those in another modality, a mechanism that underlies the near-ubiquitous phenomenon of multisensory interaction (1). This phenomenon has been the focus of a large and burgeoning theoretical and experimental literature (1-4). One feature of environmental stimuli important for adaptive behavior is their rewarding or rewardpredicting properties. Surprisingly, we know little about how reward associations in one sensory modality influence processing in other modlities. In particular, whether an association with increased reward, known to increase the accuracy of perceptual processing within that sensory modality (5-10), can transfer between modalities for concurrently presented stimuli is unclear. This is important both because of what it reveals about the nature of cross-modal associations and because it might constitute an important perceptual mechanism in its own right. For example, one can easily imagine the importance of increased sensitivity to change in the visual scene to the concurrent sound of a tiger roaring or the mating call of a conspecific.To study cross-modal transfer of value, we designed a visual orientation discrimination task in which visual stimuli were presented concurrently with one of two arbitrary pure tones previously associated with different levels of monetary reward. Critically, these tones were task-irrelevant and bore no relationship either to the orientation of the visual stimulus or to the outcome of the trial.(No feedback was presented about the accuracy of perceptual judgments, and performance on the orientation discrimination task was not related to the payment subjects...

show abstract

“…Visual sensitivity to coherent human motion in point-light displays is also enhanced by auditory cues that are meaningful and synchronous. For example, the presentation of synchronous tap sounds increases visual sensitivity to coherent point-light displays of tapping feet presented within masks (Arrighi, Marini, & Burr, 2009). Because the audio cues in this study were always synchronous with and meaningfully related to the visual cues, it remains to be determined whether the meaningful relationship, per se, between auditory and visual information enhances visual sensitivity to human movement ( Figure 1).…”

Section: Introductionmentioning

confidence: 85%

I can see you better if I can hear you coming: Action-consistent sounds facilitate the visual detection of human gait

Thomas

Shiffrar

2010

Journal of Vision

View full text Add to dashboard Cite

Observers are remarkably sensitive to point-light displays of human movement. The Superior Temporal Sulcus (STS) and premotor cortex are implicated in the visual perception of point-light human actions and the integration of perceptual signals across modalities. These neurophysiological findings suggest that auditory information might impact visual sensitivity to point-light displays of human actions. Previous research has demonstrated that coincident, action-consistent sounds enhance visual sensitivity to the presence of coherent point-light displays of human movement. Here we ask whether visual detection sensitivity is modulated specifically by the meaningfulness of sounds that are coincident with observed point-light actions. To test this hypothesis, two psychophysical studies were conducted wherein participants detected the presence of coherent point-light walkers in a mask under unimodal or audiovisual conditions. Participants in audiovisual conditions heard either tones or actual footfalls coincident with the seen walkers' footsteps. Detection sensitivity increased when visual displays were paired with veridical auditory cues (footfalls), but not when paired with simple tones. The footfall advantage disappeared when the visual stimuli were inverted. These results suggest that the visual system makes use of auditory cues during the visual analysis of human action when there is a meaningful match between the auditory and visual cues.

show abstract

Meaningful auditory information enhances perception of visual biological motion

Cited by 41 publications

References 35 publications

Music can elicit a visual motion aftereffect

Music can elicit a visual motion aftereffect

Cross-modal effects of value on perceptual acuity and stimulus encoding

I can see you better if I can hear you coming: Action-consistent sounds facilitate the visual detection of human gait

Contact Info

Product

Resources

About