Perception of intersensory synchrony: A tutorial review

Vroomen, Jean; Keetels, Mirjam

doi:10.3758/app.72.4.871

Cited by 386 publications

(410 citation statements)

References 114 publications

Supporting

Mentioning

390

Contrasting

Unclassified

Order By: Relevance

“…The PSS corresponds to the AV asynchrony at which an individual responds at chance level (50%) in a TOJ task and thus taken as a true subjective simultaneity estimate (Vroomen and Keetels, 2010).…”

Section: Psychophysics -Point Of Subjective Simultaneitymentioning

confidence: 99%

Encoding of event timing in the phase of neural oscillations

2014

View full text Add to dashboard Cite

a b s t r a c t a r t i c l e i n f oTime perception is a critical component of conscious experience. To be in synchrony with the environment, the brain must deal not only with differences in the speed of light and sound but also with its computational and neural transmission delays. Here, we asked whether the brain could actively compensate for temporal delays by changing its processing time. Specifically, can changes in neural timing or in the phase of neural oscillation index perceived timing? For this, a lag-adaptation paradigm was used to manipulate participants' perceived audiovisual (AV) simultaneity of events while they were recorded with magnetoencephalography (MEG). Desynchronized AV stimuli were presented rhythmically to elicit a robust 1 Hz frequency-tagging of auditory and visual cortical responses. As participants' perception of AV simultaneity shifted, systematic changes in the phase of entrained neural oscillations were observed. This suggests that neural entrainment is not a passive response and that the entrained neural oscillation shifts in time. Crucially, our results indicate that shifts in neural timing in auditory cortices linearly map participants' perceived AV simultaneity. To our knowledge, these results provide the first mechanistic evidence for active neural compensation in the encoding of sensory event timing in support of the emergence of time awareness.

show abstract

Section: Psychophysics -Point Of Subjective Simultaneitymentioning

confidence: 99%

Encoding of event timing in the phase of neural oscillations

2014

View full text Add to dashboard Cite

show abstract

“…In order to avoid difficulties in interpretation, we used stimuli of rather large duration differences in order to make it more likely that indeed the perceived onset of the individual components, and not also their offsets, is used in the subjects' judgments. Timing in multisensory events is a complex phenomenon based on multiple factors like temporal characteristics of the unimodal components, intersensory delays, and transmission differences in the different modalities (for a recent review see Vroomen and Keetels 2010). Therefore, it might not be straightforward to base predictions of stimulus duration effects in audio-visual synchrony perception on results from unimodal experiments.…”

Section: Introductionmentioning

confidence: 99%

Effects of stimulus duration on audio-visual synchrony perception

et al. 2012

View full text Add to dashboard Cite

The integration of visual and auditory inputs in the human brain occurs only if the components are perceived in temporal proximity, that is, when the intermodal time difference falls within the so-called subjective synchrony range. We used the midpoint of this range to estimate the point of subjective simultaneity (PSS). We measured the PSS for audio-visual (AV) stimuli in a synchrony judgment task, in which subjects had to judge a given AV stimulus using three response categories (audio first, synchronous, video first). The relevant stimulus manipulation was the duration of the auditory and visual components. Results for unimodal auditory and visual stimuli have shown that the perceived onset shifts to relatively later positions with increasing stimulus duration. These unimodal shifts should be reflected in changing PSS values, when AV stimuli with different durations of the auditory and visual components are used. The results for 17 subjects showed indeed a significant shift of the PSS for different duration combinations of the stimulus components. Because the shifts were approximately equal for duration changes in either of the components, no net shift of the PSS was observed as long as the durations of the two components were equal. This result indicates the need to appropriately account for unimodal timing effects when quantifying intermodal synchrony perception.

show abstract

“…The temporal integration experiments followed a standard simultaneity judgement methodology from cognitive psychology, outlined in [41]. Audiovisual sequences were presented successively with a short response period in-between, consistent with the "Absolute Category Rating" method recommended by the ITU-T [17].…”

Section: Stimuli and Proceduresmentioning

confidence: 99%

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Griwodz

Halvorsen

et al. 2014

Multimed Tools Appl

View full text Add to dashboard Cite

Rules-of-thumb for noticeable and detrimental asynchrony between audio and video streams have long since been established from the contributions of several studies. Although these studies share similar findings, none have made any discernible assumptions regarding audio and video quality. Considering the use of active adaptation in present and upcoming streaming systems, audio and video will continue to be delivered in separate streams; consequently, the assumption that the rules-of-thumb hold independent of quality needs to be challenged. To put this assumption to the test, we focus on the detection, not the appraisal, of asynchrony at different levels of distortion. Cognitive psychologists use the term temporal integration to describe the failure to detect asynchrony. The term refers to a perceptual process with an inherent buffer for short asynchronies, where corresponding auditory and visual signals are merged into one experience. Accordingly, this paper discusses relevant causes and concerns with regards to asynchrony, it introduces research on audiovisual perception, and it moves on to explore the impact of audio and video quality on the temporal integration of different audiovisual events. Three content types are explored, speech from a news broadcast, music presented by a drummer, and physical action in the Multimed Tools Appl (2015) 74:345-365 form of a chess game. Within these contexts, we found temporal integration to be very robust to quality discrepancies between the two modalities. In fact, asynchrony detection thresholds varied considerably more between the different content than they did between distortion levels. Nevertheless, our findings indicate that the assumption concerning the independence of asynchrony and audiovisual quality may have to be reconsidered.

show abstract

Perception of intersensory synchrony: A tutorial review

Cited by 386 publications

References 114 publications

Encoding of event timing in the phase of neural oscillations

Encoding of event timing in the phase of neural oscillations

Effects of stimulus duration on audio-visual synchrony perception

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Contact Info

Product

Resources

About