Processing of audiovisual associations in the human brain: dependency on expectations and rule complexity

Lindström, Riitta; Paavilainen, Petri; Kujala, Teija; Tervaniemi, Mari

doi:10.3389/fpsyg.2012.00159

Cited by 13 publications

(23 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, in cued reaction time tasks, the largest decrease in reaction times is typically found when there is a constant delay between the cue and the target, and this advantage is reduced as the delay becomes more variable (Niemi & Näätänen, 1981). Moreover, there is a crucial temporal window during which audiovisual stimuli are integrated (Lindström, Paavilainen, Kujala, & Tervaniemi, 2012; Van Atteveldt, Formisano, Blomert, & Goebel, 2007; Van Wassenhove et al, 2007; Zampini, Shore, & Spence, 2003) and although the width of the window varies, the point of maximal integration is consistently when visual stimuli precede auditory stimuli (Thorne & Debener, 2008; Van Wassenhove et al, 2007). Similarly, electrophysiological recordings show an enhancement of the neural response to auditory tones when they are preceded by a somatosensory or visual stimulus (Lakatos, Chen, O'Connell, Mills, & Schroeder, 2007; Kayser & Logothetis, 2009; Lakatos et al, 2009; Thorne, De Vos, Viola, & Debener, 2011; Wallace, Wilkinson, & Stein, 1996), with the largest AV effect found at an audiovisual SOA of ~65 ms.…”

Section: Discussionmentioning

confidence: 99%

Rhythmicity and cross-modal temporal cues facilitate detection

et al. 2014

View full text Add to dashboard Cite

Temporal structure in the environment often has predictive value for anticipating the occurrence of forthcoming events. In this study we investigated the influence of two types of predictive temporal information on the perception of near-threshold auditory stimuli: 1) intrinsic temporal rhythmicity within an auditory stimulus stream and 2) temporally-predictive visual cues. We hypothesized that combining predictive temporal information within- and across-modality should decrease the threshold at which sounds are detected, beyond the advantage provided by each information source alone. Two experiments were conducted in which participants had to detect tones in noise. Tones were presented in either rhythmic or random sequences and were preceded by a temporally predictive visual signal in half of the trials. We show that detection intensities are lower for rhythmic (vs. random) and audiovisual (vs. auditory-only) presentation, independent from response bias, and that this effect is even greater for rhythmic audiovisual presentation. These results suggest that both types of temporal information are used to optimally process sounds that occur at expected points in time (resulting in enhanced detection), and that multiple temporal cues are combined to improve temporal estimates. Our findings underscore the flexibility and proactivity of the perceptual system which uses within- and across-modality temporal cues to anticipate upcoming events and process them optimally.

show abstract

Section: Discussionmentioning

confidence: 99%

Rhythmicity and cross-modal temporal cues facilitate detection

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Visual material can establish predictions for a sound (Bendixen et al, 2012 ; Lindström et al, 2012 ; Clark, 2013 ). Particularly, a mismatch between a predictive note-like symbol and the pitch of the corresponding sound elicits brain responses that signal the violation of a prediction.…”

Section: Experiments 1: Separate Predictionsmentioning

confidence: 99%

“…The IR presumably reflects the prediction error at sensory levels of processing. At cognitive levels, where the sound is categorized with respect to task affordances (e.g., whether or which button has to be pressed), the N2b is elicited (Widmann et al, 2004 ; Lindström et al, 2012 ). The fronto-centrally distributed ERP component is observable at approximately 200 ms after sound onset.…”

Section: Experiments 1: Separate Predictionsmentioning

confidence: 99%

“…N2b is usually elicited by task-relevant deviants in oddball paradigms (Ritter et al, 1979 ; Näätänen et al, 1982 ; Novak et al, 1990 ) and in visual-auditory paradigms with a response task (cf. Lindström et al, 2012 ; Pieszek et al, 2013 ; but Tervaniemi et al, 2003 : no response was required). Functionally, the N2b is assumed to indicate a “decision process related to sensory discrimination of attended stimuli,” i.e., attentive target detection (Ritter et al, 1979 , p. 1360).…”

Section: Experiments 1: Separate Predictionsmentioning

confidence: 99%

See 1 more Smart Citation

Separate and concurrent symbolic predictions of sound features are processed differently

Pieszek¹,

Schröger²,

Widmann³

2014

Front. Psychol.

View full text Add to dashboard Cite

The studies investigated the impact of predictive visual information about the pitch and location of a forthcoming sound on the sound processing. In Symbol-to-Sound matching paradigms, symbols induced predictions of particular sounds. The brain's error signals (IR and N2b components of the event-related potential) were measured in response to occasional violations of the prediction, i.e., when a sound was incongruent to the corresponding symbol. IR and N2b index the detection of prediction violations at different levels, IR at a sensory and N2b at a cognitive level. Participants evaluated the congruency between prediction and actual sound by button press. When the prediction referred to only the pitch or only the location feature (Experiment 1), the violation of each feature elicited IR and N2b. The IRs to pitch and location violations revealed differences in the in time course and topography, suggesting that they were generated in feature-specific sensory areas. When the prediction referred to both features concurrently (Experiment 2), that is, the symbol predicted the sound's pitch and location, either one or both predictions were violated. Unexpectedly, no significant effects in the IR range were obtained. However, N2b was elicited in response to all violations. N2b in response to concurrent violations of pitch and location had a shorter latency. We conclude that associative predictions can be established by arbitrary rule-based symbols and for different sound features, and that concurrent violations are processed in parallel. In complex situations as in Experiment 2, capacity limitations appear to affect processing in a hierarchical manner. While predictions were presumably not reliably established at sensory levels (absence of IR), they were established at more cognitive levels, where sounds are represented categorially (presence of N2b).

show abstract

“…The neural mechanisms of audiovisual integration can be investigated with the N2 response, a negative deflection of the eventrelated potential (ERP) during the early phases of auditory change detection (Näätänen, Simpson, & Loveless, 1982;Novak, Ritter, Vaughan, & Wiznitzer, 1990). N2 response is a feasible tool for investigating how visual information affects auditory processing at the cortical level (Lindström, Paavilainen, Kujala, & Tervaniemi, 2012). N2 response to attended auditory stimulus changes consists of two components: mismatch negativity (MMN) and N2b (Näätänen, 1992;Näätänen, Gaillard, & Mäntysalo, 1978;Näätänen et al, 1982;Novak et al, 1990).…”

mentioning

confidence: 99%

Audiovisual attention boosts letter‐speech sound integration

et al. 2013

View full text Add to dashboard Cite

We studied attention effects on the integration of written and spoken syllables in fluent adult readers by using event-related brain potentials. Auditory consonant-vowel syllables, including consonant and frequency changes, were presented in synchrony with written syllables or their scrambled images. Participants responded to longer-duration auditory targets (auditory attention), longer-duration visual targets (visual attention), longer-duration auditory and visual targets (audiovisual attention), or counted backwards mentally. We found larger negative responses for spoken consonant changes when they were accompanied by written syllables than when they were accompanied by scrambled text. This effect occurred at an early latency (∼ 140 ms) during audiovisual attention and later (∼ 200 ms) during visual attention. Thus, audiovisual attention boosts the integration of speech sounds and letters.

show abstract

Processing of audiovisual associations in the human brain: dependency on expectations and rule complexity

Cited by 13 publications

References 18 publications

Rhythmicity and cross-modal temporal cues facilitate detection

Rhythmicity and cross-modal temporal cues facilitate detection

Separate and concurrent symbolic predictions of sound features are processed differently

Audiovisual attention boosts letter‐speech sound integration

Contact Info

Product

Resources

About