Predictability affects the perception of audiovisual synchrony in complex sequences

Cook, Laura; Valkenburg, David L. Van; Badcock, David R.

doi:10.3758/s13414-011-0185-8

Cited by 7 publications

(4 citation statements)

References 39 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cook et al (2011) recently compared synchrony perception for tones and flashes that were equally spaced in time but appeared in either a predictable order (alternating high and low tones in two auditory streams which were simultaneously ascending or descending in pitch) or a less predictable order (a scrambled version of this sequence). They found that observers perceived audio-visual synchrony over wider time windows for the more unpredictable streams.…”

Section: Discussionmentioning

confidence: 99%

Temporal Structure and Complexity Affect Audio-Visual Correspondence Detection

Denison¹,

Driver²,

Ruff³

2013

Front. Psychology

View full text Add to dashboard Cite

Synchrony between events in different senses has long been considered the critical temporal cue for multisensory integration. Here, using rapid streams of auditory and visual events, we demonstrate how humans can use temporal structure (rather than mere temporal coincidence) to detect multisensory relatedness. We find psychophysically that participants can detect matching auditory and visual streams via shared temporal structure for crossmodal lags of up to 200 ms. Performance on this task reproduced features of past findings based on explicit timing judgments but did not show any special advantage for perfectly synchronous streams. Importantly, the complexity of temporal patterns influences sensitivity to correspondence. Stochastic, irregular streams – with richer temporal pattern information – led to higher audio-visual matching sensitivity than predictable, rhythmic streams. Our results reveal that temporal structure and its complexity are key determinants for human detection of audio-visual correspondence. The distinctive emphasis of our new paradigms on temporal patterning could be useful for studying special populations with suspected abnormalities in audio-visual temporal perception and multisensory integration.

show abstract

Section: Discussionmentioning

confidence: 99%

Temporal Structure and Complexity Affect Audio-Visual Correspondence Detection

Denison¹,

Driver²,

Ruff³

2013

Front. Psychology

View full text Add to dashboard Cite

show abstract

“…The amount of auditory and visual information available during ensemble performance is immense, and to make sense of it musicians must be able to bind together signals produced by the same event. Perceptual integration of auditory and visual signals is more likely to occur when those signals share temporal or semantic properties (Arrighi et al, 2006 ; Petrini et al, 2010a ; Cook et al, 2011 ). People tolerate some asynchrony between auditory and visual signals.…”

Section: Introductionmentioning

confidence: 99%

“…In the short-term, several minutes' exposure to a particular degree of audiovisual asynchrony can prompt recalibration of sensory processing mechanisms, altering observers' perceptions of synchrony (Fujisaki et al, 2004 ; Vroomen et al, 2004 ). In the longer term, observers' sensitivity to audiovisual asynchrony relates to their familiarity with the presented stimuli (Vatakis and Spence, 2006 ; Saygin et al, 2008 ) and how readily they can predict the events in a perceived stimulus sequence (Petrini et al, 2009b ; Cook et al, 2011 ; Lee and Noppeney, 2011 ). For instance, when presented with point-light displays of drumming actions containing arm but no drumstick markers, skilled percussionists are able to detect audiovisual asynchronies, while novices are not (Petrini et al, 2009b ).…”

Section: Introductionmentioning

confidence: 99%

Context-specific effects of musical expertise on audiovisual integration

Bishop

Goebl

2014

Front. Psychol.

View full text Add to dashboard Cite

Ensemble musicians exchange auditory and visual signals that can facilitate interpersonal synchronization. Musical expertise improves how precisely auditory and visual signals are perceptually integrated and increases sensitivity to asynchrony between them. Whether expertise improves sensitivity to audiovisual asynchrony in all instrumental contexts or only in those using sound-producing gestures that are within an observer's own motor repertoire is unclear. This study tested the hypothesis that musicians are more sensitive to audiovisual asynchrony in performances featuring their own instrument than in performances featuring other instruments. Short clips were extracted from audio-video recordings of clarinet, piano, and violin performances and presented to highly-skilled clarinetists, pianists, and violinists. Clips either maintained the audiovisual synchrony present in the original recording or were modified so that the video led or lagged behind the audio. Participants indicated whether the audio and video channels in each clip were synchronized. The range of asynchronies most often endorsed as synchronized was assessed as a measure of participants' sensitivities to audiovisual asynchrony. A positive relationship was observed between musical training and sensitivity, with data pooled across stimuli. While participants across expertise groups detected asynchronies most readily in piano stimuli and least readily in violin stimuli, pianists showed significantly better performance for piano stimuli than for either clarinet or violin. These findings suggest that, to an extent, the effects of expertise on audiovisual integration can be instrument-specific; however, the nature of the sound-producing gestures that are observed has a substantial effect on how readily asynchrony is detected as well.

show abstract

“…In this way, temporal integration can be explored for simple stimuli such as lights and tones [6], but also for more complex scenarios such as speech [42]. Several studies have looked at the variations in temporal integration for different events, supplying new stock to an ongoing debate on the special nature of speech.…”

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Griwodz

Halvorsen

et al. 2014

Multimed Tools Appl

View full text Add to dashboard Cite

Rules-of-thumb for noticeable and detrimental asynchrony between audio and video streams have long since been established from the contributions of several studies. Although these studies share similar findings, none have made any discernible assumptions regarding audio and video quality. Considering the use of active adaptation in present and upcoming streaming systems, audio and video will continue to be delivered in separate streams; consequently, the assumption that the rules-of-thumb hold independent of quality needs to be challenged. To put this assumption to the test, we focus on the detection, not the appraisal, of asynchrony at different levels of distortion. Cognitive psychologists use the term temporal integration to describe the failure to detect asynchrony. The term refers to a perceptual process with an inherent buffer for short asynchronies, where corresponding auditory and visual signals are merged into one experience. Accordingly, this paper discusses relevant causes and concerns with regards to asynchrony, it introduces research on audiovisual perception, and it moves on to explore the impact of audio and video quality on the temporal integration of different audiovisual events. Three content types are explored, speech from a news broadcast, music presented by a drummer, and physical action in the Multimed Tools Appl (2015) 74:345-365 form of a chess game. Within these contexts, we found temporal integration to be very robust to quality discrepancies between the two modalities. In fact, asynchrony detection thresholds varied considerably more between the different content than they did between distortion levels. Nevertheless, our findings indicate that the assumption concerning the independence of asynchrony and audiovisual quality may have to be reconsidered.

show abstract

Predictability affects the perception of audiovisual synchrony in complex sequences

Cited by 7 publications

References 39 publications

Temporal Structure and Complexity Affect Audio-Visual Correspondence Detection

Temporal Structure and Complexity Affect Audio-Visual Correspondence Detection

Context-specific effects of musical expertise on audiovisual integration

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Contact Info

Product

Resources

About