Computational Requirements and Synchronization Issues for Virtual Acoustic Displays

Miner, Nadine E.; Caudell, Thomas P.

doi:10.1162/105474698565802

Cited by 27 publications

(12 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the object-action videos, however, the participants only tolerated an auditory lag of 188 msec and a visual lag of 75 msec. These data suggest that participants are less sensitive to asynchrony in audiovisual speech, a result that has also been obtained by other studies (Hollier & Rimell, 1998;Miner & Caudell, 1998). Note, however, that for several reasons 1 these early studies might have overestimated peopleʼs tolerance for asynchrony in audiovisual speech processing.…”

Section: Introductionsupporting

confidence: 84%

Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

Obermeier

Gunter

2015

Journal of Cognitive Neuroscience

View full text Add to dashboard Cite

This experiment investigates the integration of gesture and speech from a multisensory perspective. In a disambiguation paradigm, participants were presented with short videos of an actress uttering sentences like "She was impressed by the BALL, because the GAME/DANCE...." The ambiguous noun (BALL) was accompanied by an iconic gesture fragment containing information to disambiguate the noun toward its dominant or subordinate meaning. We used four different temporal alignments between noun and gesture fragment: the identification point (IP) of the noun was either prior to (+120 msec), synchronous with (0 msec), or lagging behind the end of the gesture fragment (-200 and -600 msec). ERPs triggered to the IP of the noun showed significant differences for the integration of dominant and subordinate gesture fragments in the -200, 0, and +120 msec conditions. The outcome of this integration was revealed at the target words. These data suggest a time window for direct semantic gesture-speech integration ranging from at least -200 up to +120 msec. Although the -600 msec condition did not show any signs of direct integration at the homonym, significant disambiguation was found at the target word. An explorative analysis suggested that gesture information was directly integrated at the verb, indicating that there are multiple positions in a sentence where direct gesture-speech integration takes place. Ultimately, this would implicate that in natural communication, where a gesture lasts for some time, several aspects of that gesture will have their specific and possibly distinct impact on different positions in an utterance.

show abstract

Section: Introductionsupporting

confidence: 84%

Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

Obermeier

Gunter

2015

Journal of Cognitive Neuroscience

View full text Add to dashboard Cite

show abstract

“…Several studies have looked at the variations in temporal integration for different events, supplying new stock to an ongoing debate on the special nature of speech. A majority of the studies has found the perception of synchrony in speech to be more tolerant to temporal offsets compared to isolated actions that typically have anticipatory moments of impact, such as a hammer hitting a peg [8], or other encounters between tools and objects [28,36]. The robustness of intelligible speech has also been demonstrated in comparison to monkey calls [39] and non-native languages [29].…”

Section: Temporal Integration and Quality Distortionmentioning

confidence: 99%

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Griwodz

Halvorsen

et al. 2014

Multimed Tools Appl

View full text Add to dashboard Cite

Rules-of-thumb for noticeable and detrimental asynchrony between audio and video streams have long since been established from the contributions of several studies. Although these studies share similar findings, none have made any discernible assumptions regarding audio and video quality. Considering the use of active adaptation in present and upcoming streaming systems, audio and video will continue to be delivered in separate streams; consequently, the assumption that the rules-of-thumb hold independent of quality needs to be challenged. To put this assumption to the test, we focus on the detection, not the appraisal, of asynchrony at different levels of distortion. Cognitive psychologists use the term temporal integration to describe the failure to detect asynchrony. The term refers to a perceptual process with an inherent buffer for short asynchronies, where corresponding auditory and visual signals are merged into one experience. Accordingly, this paper discusses relevant causes and concerns with regards to asynchrony, it introduces research on audiovisual perception, and it moves on to explore the impact of audio and video quality on the temporal integration of different audiovisual events. Three content types are explored, speech from a news broadcast, music presented by a drummer, and physical action in the Multimed Tools Appl (2015) 74:345-365 form of a chess game. Within these contexts, we found temporal integration to be very robust to quality discrepancies between the two modalities. In fact, asynchrony detection thresholds varied considerably more between the different content than they did between distortion levels. Nevertheless, our findings indicate that the assumption concerning the independence of asynchrony and audiovisual quality may have to be reconsidered.

show abstract

“…An intuitive way to investigate the effect of expertise on multisensory integration of action is to measure changes in sensitivity to audiovisual asynchrony (Arrighi, Alais, & Burr, 2006;Hollier, Rimell, Hands, & Voelcker, 1999;Miner & Caudell, 1998;van Wassenhove, Grant, & Poeppel, 2007;Vatakis & Spence, 2006a, 2006b. By altering the degree of asynchrony between the visual and auditory stimuli, we obtain an indirect measure of the changes in the neural tolerance of audiovisual asynchrony, and can apply this measure to study expertise.…”

Section: Visual and Visuomotor Experience In Action Observationmentioning

confidence: 99%

Experience and the Perception of Biological Motion

Pollick¹,

Jola²,

Petrini³

et al. 2012

People Watching

View full text Add to dashboard Cite

Many discussions of biological motion perception involve a description of observers’ attunements for recognizing gender, emotion, action, and identity from point-light displays. This chapter describes an often-neglected determinant of biological motion perception: the role of expertise. First, the authors describe how variability among observers is essential for developing a comprehensive theory of biological motion perception. Then, they describe how the distributed network of brain areas devoted to biological motion perception provides an opportunity for this brain network to be applied to new tasks and environments.

show abstract

Computational Requirements and Synchronization Issues for Virtual Acoustic Displays

Cited by 27 publications

References 9 publications

Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Experience and the Perception of Biological Motion

Contact Info

Product

Resources

About