Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli

Vatakis, Argiro; Spence, Charles

doi:10.3758/bf03193776

Cited by 263 publications

(284 citation statements)

References 59 publications

(95 reference statements)

Supporting

Mentioning

252

Contrasting

Unclassified

Order By: Relevance

“…The influence of such expectations has been demonstrated, e.g. by more erroneous temporal-order judgements on gender matching than mismatching AV speech clips (Vatakis and Spence 2007). Multisensory speech congruence detection might be particularly sensitive to expectations, as suggested by results indicating that the McGurk illusion does not occur if participants interpret the sounds as noise (Tuomainen et al 2005; Figure 1B-C).…”

Section: Role Of the Observermentioning

confidence: 86%

“…van Atteveldt et al 2007). These processing enhancements might likewise arise from the expectationsthat congruent crossmodal signals likely share a common source (Vatakis and Spence 2007). The co-activation and expectation mechanisms are not mutually exclusive ( Figure 1A).Notably, the detection of congruence across certain perceptual features of multisensory stimuli might sometimes have a more hardwired nature, which is possibly based on the properties of receptive fields of multisensory neurons ( Figure 1A).…”

Section: Stimulus-based Effectsmentioning

confidence: 99%

See 1 more Smart Citation

The COGs (context, object, and goals) in multisensory processing

et al. 2016

View full text Add to dashboard Cite

Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensoryprocesses and "top-down" control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topicshave been traditionally studied separately, the mechanisms orchestrating real-world multisensory processingremain unclear.Past work has revealed that the observer's goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes mightoccur independently of these goals. Consequently, other forms of top-down control beyond goaldependence are necessaryto explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control includesensitivity to stimulus context as well as the detection of matches(or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object-and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing, and discuss their possible broader implications. Attention, multisensory, control, object, audiovisual, brain mapping 3 Research from the past 30 years has demonstrated a whole range of behavioural benefits engendered by integrating information across the senses (multisensory integration, MSI), including faster motor responses and facilitated object recognition in noisy environments (e.g. Stein 2012). In a separate and independent manner, studies employing unisensory stimuli have been critically advancing our understanding of the nature and the importance of top-down mechanisms that control information processing. Thetopdown nature of these mechanisms lies in that they shape perceptual processing of new inputs by activating information stored in higher-order brain areas (e.g. Summerfield and Egner 2009). Key words:Studies of top-down control have traditionally focused on attentional (i.e. goaldependent)mechanisms, which promote the processing of stimulior objects in the environment that areimportant tothe current behavioural goals of the observer (e.g.Desimone and Duncan 1995). These mechanisms enhance the processing of stimuli appearing in task-relevant spatial locations and/or moments in time. These mechanisms likewise facilitate the processingof those stimuli whose attributes (e.g. colour red), feature dimensions (e.g. shape) or identity (e.g. a particular face) match the observer's goals.Simultaneously, it has been increasingly recognised in the literature that information processing issensitive to other types oftop-down processes, principally those gauged by thememory of past stimulation and one's expectations (see Figure 1A for a summar...

show abstract

Section: Role Of the Observermentioning

confidence: 86%

Section: Stimulus-based Effectsmentioning

confidence: 99%

The COGs (context, object, and goals) in multisensory processing

et al. 2016

View full text Add to dashboard Cite

show abstract

“…When judging the temporal order of the individual components of facevoice pairings, performance is worse when the genders of faces and voices agree. Vatakis and Spence (2007) explain that weakened performance indicates integration, which inhibits comparative analysis of the individual components. This identity cue does not seem to be triggered under all conditions.…”

Section: Causality and Unitymentioning

confidence: 99%

Causality and cross-modal integration.

Schutz

Kubovy

2009

Journal of Experimental Psychology: Human Perception and Perfor

View full text Add to dashboard Cite

Schutz and Lipscomb (2007) reported an audiovisual illusion in which the length of the gesture used to produce a sound altered the perception of that sound's duration. This contradicts the widely accepted claim that the auditory system generally dominates temporal tasks because of its superior temporal acuity. Here, in the first of 4 experiments, we show that impact gestures influence duration ratings of percussive but not sustained sounds. In the 2nd, we show that the illusion is present even if the percussive sound occurs up to 700 ms after the visible impact, but disappears if the percussive sound precedes the visible impact. In the 3rd experiment, we show that only the motion after the visible impact influences perceived tone duration. The 4th experiment (replacing the impact gestures with the written text long and short) suggests that the phenomenon is not due to response bias. Given that visual influence in this paradigm is dependent on the presence of an ecologically plausible audiovisual relationship, we conclude that cross-modal causality plays a key role in governing the integration of sensory information.

show abstract

“…There is evidence for an effect of semantic congruency, which supports the unity assumption, using speech stimuli. For example, Vatakis and Spence (2007) found that participants were worse at making temporal order judgments between sound and vision for speech video when the gender was matched across modalities as opposed to when one was male and one female. They interpreted this finding to be due to the mismatched stimuli being less susceptible to temporal ventriloquism since they were less likely to originate from the same source.…”

Section: Discussionmentioning

confidence: 99%

Predictability affects the perception of audiovisual synchrony in complex sequences

Cook

Valkenburg

Badcock

2011

Atten Percept Psychophys

View full text Add to dashboard Cite

The ability to make accurate audiovisual synchrony judgments is affected by the "complexity" of the stimuli: We are much better at making judgments when matching single beeps or flashes as opposed to video recordings of speech or music. In the present study, we investigated whether the predictability of sequences affects whether participants report that auditory and visual sequences appear to be temporally coincident. When we reduced their ability to predict both the next pitch in the sequence and the temporal pattern, we found that participants were increasingly likely to report that the audiovisual sequences were synchronous. However, when we manipulated pitch and temporal predictability independently, the same effect did not occur. By altering the temporal density (items per second) of the sequences, we further determined that the predictability effect occurred only in temporally dense sequences: If the sequences were slow, participants' responses did not change as a function of predictability. We propose that reduced predictability affects synchrony judgments by reducing the effective pitch and temporal acuity in perception of the sequences.

show abstract

Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli

Cited by 263 publications

References 59 publications

The COGs (context, object, and goals) in multisensory processing

The COGs (context, object, and goals) in multisensory processing

Causality and cross-modal integration.

Predictability affects the perception of audiovisual synchrony in complex sequences

Contact Info

Product

Resources

About