Temporal window of integration in auditory-visual speech perception

Wassenhove, Virginie van; Grant, Ken W.; Poeppel, David

doi:10.1016/j.neuropsychologia.2006.01.001

Cited by 511 publications

(558 citation statements)

References 47 publications

Supporting

Mentioning

493

Contrasting

Unclassified

Order By: Relevance

“…Like previous works have shown [21,26,42], asynchrony can be detected at very short thresholds, likely depending on the nature of the audiovisual event. Our observations indicate that such caution would involve the avoidance of video stream delays entirely, since the windows of temporal integration for two of three content types do not extend to audio lead asynchrony, irrespective of distortion.…”

Section: Resultssupporting

confidence: 55%

“…In this way, temporal integration can be explored for simple stimuli such as lights and tones [6], but also for more complex scenarios such as speech [42]. Several studies have looked at the variations in temporal integration for different events, supplying new stock to an ongoing debate on the special nature of speech.…”

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

“…1 Overview of previously published thresholds of perceived synchrony, illustrating the different measures and event types applied to the study of temporal integration. Where some ask participants to indicate the point at which an auditory and a visual stream are no longer perceived as synchronous [8,14,21,26], others use simultaneity judgements and curve fittings to calculate the full-width at half-maximum (FWHM) [5] or the 95 % confidence interval [42]. Others still use temporal order judgements and calculate the slopes of cumulative distribution functions [11] With new multimedia platforms come new approaches to efficient streaming, and with them come old concerns.…”

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

See 2 more Smart Citations

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Griwodz

Halvorsen

et al. 2014

Multimed Tools Appl

View full text Add to dashboard Cite

Rules-of-thumb for noticeable and detrimental asynchrony between audio and video streams have long since been established from the contributions of several studies. Although these studies share similar findings, none have made any discernible assumptions regarding audio and video quality. Considering the use of active adaptation in present and upcoming streaming systems, audio and video will continue to be delivered in separate streams; consequently, the assumption that the rules-of-thumb hold independent of quality needs to be challenged. To put this assumption to the test, we focus on the detection, not the appraisal, of asynchrony at different levels of distortion. Cognitive psychologists use the term temporal integration to describe the failure to detect asynchrony. The term refers to a perceptual process with an inherent buffer for short asynchronies, where corresponding auditory and visual signals are merged into one experience. Accordingly, this paper discusses relevant causes and concerns with regards to asynchrony, it introduces research on audiovisual perception, and it moves on to explore the impact of audio and video quality on the temporal integration of different audiovisual events. Three content types are explored, speech from a news broadcast, music presented by a drummer, and physical action in the Multimed Tools Appl (2015) 74:345-365 form of a chess game. Within these contexts, we found temporal integration to be very robust to quality discrepancies between the two modalities. In fact, asynchrony detection thresholds varied considerably more between the different content than they did between distortion levels. Nevertheless, our findings indicate that the assumption concerning the independence of asynchrony and audiovisual quality may have to be reconsidered.

show abstract

Section: Resultssupporting

confidence: 55%

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

See 1 more Smart Citation

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Griwodz

Halvorsen

et al. 2014

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…The sound seems to emanate from the location of the mouth. Both of these effects diminish (within limits-see van Wassenhove et al [4] regarding the McGurk effect) with asynchrony between sound and lip movements [5][6][7][8][9][10][11].…”

Section: Bindingmentioning

confidence: 99%

Multistability, cross-modal binding and the additivity of conjoined grouping principles

Kubovy

2012

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

We present a sceptical view of multimodal multistability-drawing most of our examples from the relation between audition and vision. We begin by summarizing some of the principal ways in which audio-visual binding takes place. We review the evidence that unambiguous stimulation in one modality may affect the perception of a multistable stimulus in another modality. Crossmodal influences of one multistable stimulus on the multistability of another are different: they have occurred only in speech perception. We then argue that the strongest relation between perceptual organization in vision and perceptual organization in audition is likely to be by way of analogous Gestalt laws. We conclude with some general observations about multimodality.

show abstract

“…For the incongruent presentations, audiovisual stimuli looked and sounded naturally timed as the 30 msec difference in /b/ versus /g/ onsets is well within the critical AV binding window (van Wassenhove, Grant, & Poeppel, 2007). The experiment started with a short training.…”

Section: -Procedures and Designmentioning

confidence: 99%