Audiovisual synchrony perception for music, speech, and object actions

Vatakis, Argiro; Spence, Charles

doi:10.1016/j.brainres.2006.05.078

Cited by 146 publications

(155 citation statements)

References 35 publications

Supporting

Mentioning

141

Contrasting

Order By: Relevance

“…Furthermore, temporal integration varies between audiovisual events, similar to what has been noted by others [8,36]. The three content studied have quite distinct distributions for perceived synchrony, with asynchrony detected sooner in speech than in either the chess or the drums sequence.…”

Section: Resultssupporting

confidence: 83%

“…Several studies have looked at the variations in temporal integration for different events, supplying new stock to an ongoing debate on the special nature of speech. A majority of the studies has found the perception of synchrony in speech to be more tolerant to temporal offsets compared to isolated actions that typically have anticipatory moments of impact, such as a hammer hitting a peg [8], or other encounters between tools and objects [28,36]. The robustness of intelligible speech has also been demonstrated in comparison to monkey calls [39] and non-native languages [29].…”

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

See 1 more Smart Citation

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Griwodz

Halvorsen

et al. 2014

Multimed Tools Appl

View full text Add to dashboard Cite

Rules-of-thumb for noticeable and detrimental asynchrony between audio and video streams have long since been established from the contributions of several studies. Although these studies share similar findings, none have made any discernible assumptions regarding audio and video quality. Considering the use of active adaptation in present and upcoming streaming systems, audio and video will continue to be delivered in separate streams; consequently, the assumption that the rules-of-thumb hold independent of quality needs to be challenged. To put this assumption to the test, we focus on the detection, not the appraisal, of asynchrony at different levels of distortion. Cognitive psychologists use the term temporal integration to describe the failure to detect asynchrony. The term refers to a perceptual process with an inherent buffer for short asynchronies, where corresponding auditory and visual signals are merged into one experience. Accordingly, this paper discusses relevant causes and concerns with regards to asynchrony, it introduces research on audiovisual perception, and it moves on to explore the impact of audio and video quality on the temporal integration of different audiovisual events. Three content types are explored, speech from a news broadcast, music presented by a drummer, and physical action in the Multimed Tools Appl (2015) 74:345-365 form of a chess game. Within these contexts, we found temporal integration to be very robust to quality discrepancies between the two modalities. In fact, asynchrony detection thresholds varied considerably more between the different content than they did between distortion levels. Nevertheless, our findings indicate that the assumption concerning the independence of asynchrony and audiovisual quality may have to be reconsidered.

show abstract

Section: Resultssupporting

confidence: 83%

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Griwodz

Halvorsen

et al. 2014

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…We can maintain a coherent multisensory percept across a considerable range of spatial and temporal discrepancies (1-4): For example, auditory and visual speech signals can be perceived as belonging to the same multisensory "object" over temporal windows of hundreds of milliseconds (5)(6)(7). However, such misalignment can drastically affect neuronal responses in ways that may also differ between brain regions (8)(9)(10).…”

mentioning

confidence: 99%

Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex

Perrodin

Kayser

Logothetis

et al. 2014

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

When social animals communicate, the onset of informative content in one modality varies considerably relative to the other, such as when visual orofacial movements precede a vocalization. These naturally occurring asynchronies do not disrupt intelligibility or perceptual coherence. However, they occur on time scales where they likely affect integrative neuronal activity in ways that have remained unclear, especially for hierarchically downstream regions in which neurons exhibit temporally imprecise but highly selective responses to communication signals. To address this, we exploited naturally occurring face-and voice-onset asynchronies in primate vocalizations. Using these as stimuli we recorded cortical oscillations and neuronal spiking responses from functional MRI (fMRI)-localized voice-sensitive cortex in the anterior temporal lobe of macaques. We show that the onset of the visual face stimulus resets the phase of low-frequency oscillations, and that the face-voice asynchrony affects the prominence of two key types of neuronal multisensory responses: enhancement or suppression. Our findings show a three-way association between temporal delays in audiovisual communication signals, phase-resetting of ongoing oscillations, and the sign of multisensory responses. The results reveal how natural onset asynchronies in cross-sensory inputs regulate network oscillations and neuronal excitability in the voice-sensitive cortex of macaques, a suggested animal model for human voice areas. These findings also advance predictions on the impact of multisensory input on neuronal processes in face areas and other brain regions.H ow the brain parses multisensory input despite the variable and often large differences in the onset of sensory signals across different modalities remains unclear. We can maintain a coherent multisensory percept across a considerable range of spatial and temporal discrepancies (1-4): For example, auditory and visual speech signals can be perceived as belonging to the same multisensory "object" over temporal windows of hundreds of milliseconds (5-7). However, such misalignment can drastically affect neuronal responses in ways that may also differ between brain regions (8-10). We asked how natural asynchronies in the onset of face/voice content in communication signals would affect voice-sensitive cortex, a region in the ventral "object" pathway (11) where neurons (i) are selective for auditory features in communication sounds (12)(13)(14), (ii) are influenced by visual "face" content (12), and (iii) display relatively slow and temporally variable responses in comparison with neurons in primary auditory cortical or subcortical structures (14-16).Neurophysiological studies in human and nonhuman animals have provided considerable insights into the role of cortical oscillations during multisensory conditions and for parsing speech. Cortical oscillations entrain to the slow temporal dynamics of natural sounds (17)(18)(19)(20) and are thought to reflect the excitability of local networks to sensory inputs...

show abstract

“…An intuitive way to investigate the effect of expertise on multisensory integration of action is to measure changes in sensitivity to audiovisual asynchrony (Arrighi, Alais, & Burr, 2006;Hollier, Rimell, Hands, & Voelcker, 1999;Miner & Caudell, 1998;van Wassenhove, Grant, & Poeppel, 2007;Vatakis & Spence, 2006a, 2006b. By altering the degree of asynchrony between the visual and auditory stimuli, we obtain an indirect measure of the changes in the neural tolerance of audiovisual asynchrony, and can apply this measure to study expertise.…”

Section: Visual and Visuomotor Experience In Action Observationmentioning

confidence: 99%

“…To study expertise in multisensory processing, we need a domain where it is possible to differentiate between an expert and a novice population that do not share knowledge about certain multisensory actions. We chose music since it shares many characteristics not only with speech and dance, but also with object action (Vatakis & Spence, 2006a, 2006b) and is a very effective form of social communication. Furthermore, musicians possess knowledge about certain actions that is not shared with non-musicians, and this makes music events a perfect tool to study differences between experts and non-experts.…”

Section: Visual and Visuomotor Experience In Action Observationmentioning

confidence: 99%

Experience and the Perception of Biological Motion

Pollick¹,

Jola²,

Petrini³

et al. 2012

People Watching

View full text Add to dashboard Cite

Many discussions of biological motion perception involve a description of observers’ attunements for recognizing gender, emotion, action, and identity from point-light displays. This chapter describes an often-neglected determinant of biological motion perception: the role of expertise. First, the authors describe how variability among observers is essential for developing a comprehensive theory of biological motion perception. Then, they describe how the distributed network of brain areas devoted to biological motion perception provides an opportunity for this brain network to be applied to new tasks and environments.

show abstract

Audiovisual synchrony perception for music, speech, and object actions

Cited by 146 publications

References 35 publications

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex

Experience and the Perception of Biological Motion

Contact Info

Product

Resources

About