Audio-visual spatial alignment improves integration in the presence of a competing audio-visual stimulus

Fleming, Justin T.; Noyce, Abigail; Shinn‐Cunningham, Barbara

doi:10.1016/j.neuropsychologia.2020.107530

Cited by 18 publications

(13 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To summarise, in the real world, attention should be captured more strongly by stimuli that are unpredictable (Schroeder et al 2015), but also by those unknown or without a clear meaning. On the other hand, stimuli with high strong spatial and/or temporal alignment across senses (and so stronger bottom-up salience) may be more resistant to such goal-based attentional control (suppression), as we have shown here (multisensory enhancement of attentional capture; see also Santangelo & Spence 2007;Matusz & Eimer 2011;van der Burg et al 2011;Turoman et al 2021;Fleming et al 2020). As multisensory distractors captured attention more strongly even in current, context-rich settings, this confirms the importance of multisensory salience as a source of potential bottom-up attentional control in naturalistic environments (see SOMs for a short discussion of this replication).…”

Section: How We Pay Attention In Naturalistic Settingsmentioning

confidence: 54%

See 1 more Smart Citation

Towards understanding how we pay attention in naturalistic visual search settings

Turoman

Tivadar

Retsa

et al. 2020

Preprint

View full text Add to dashboard Cite

Traditional research on attentional control has largely focused on single senses and the importance of one s behavioural goals in controlling attentional selection, thus limiting its generalizability to real-world contexts. These contexts are both inherently multisensory and contain regularities that also contribute to attentional control. To get a better understanding of how attention is controlled in the real world, we investigated how visual attentional capture was impacted by top-down goals (indexed by task-set contingent attentional capture) and the multisensory nature of stimuli, as well as top-down contextual factors such as semantic relationships and temporal predictability of stimulus onset. Participants performed a multisensory version of Folk et al. (1992) spatial cueing paradigm, while their 129-channel event-related potentials (ERPs) were recorded. Reaction-time spatial cueing served as a behavioural measure of attentional control, while the N2pc ERP component was analysed both canonically and using a multivariate electrical neuroimaging (EN) framework. Behaviourally, target-congruent colour distractors captured attention more strongly when they were simultaneous than semantically congruent (nontarget-congruent colour distractors failed to capture attention), with no behavioural evidence for context modulating multisensory enhancements of capture. However, our EN analyses revealed context-based influences on attention to both visual and multisensory distractors, on how strongly they activated brain networks and in the type of activated brain networks. In both cases, these context-driven brain response modulations occurred early on (long before the traditional N2pc time-window), with network-based modulations at app. 30ms post-distractor, followed by strength-based modulations at app. 100ms post-distractor. Our findings revealed that in naturalistic settings, meaning, next to predictions (spatial, temporal etc.) might be a second important source of contextual information utilised to facilitate goal-directed attention. Therein, attentional selection is controlled by an interaction of one s goals, stimulus perceptual (multisensory-driven) salience and an interaction of stimulus meaning and its predictability. Our study demonstrates how investigating more traditional, lab-studied control mechanisms and processes more typical for everyday life reveals a complex interplay between goal-, stimulus- and context-based processes in attentional control. As such, our findings call for a revision of traditional models of visual attentional control to account for the role of both contextual and multisensory control mechanisms.

show abstract

Section: How We Pay Attention In Naturalistic Settingsmentioning

confidence: 54%

“…Also, multisensory processes are increasingly recognised as an important source of bottom-up, attentional control (e.g. Spence & Santangelo 2007; Matusz & Eimer 2011; Matusz et al, 2019a; Fleming et al, 2020). By studying these processes largely in isolation, researchers clarified how they support goal-directed behaviour.…”

Section: Discussionmentioning

confidence: 99%

Towards understanding how we pay attention in naturalistic visual search settings

Turoman

Tivadar

Retsa

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…However, in a version of the task with two competing AV stimuli, search benefits and ERP signatures of integration did depend on spatial alignment between the auditory and visual components of each stimulus. 20 Taken together, these findings indicate that in relatively simple scenes lacking in multisensory competition, temporal coherence alone is sufficient to drive AV integration. When the sensory environment becomes more complex, however, AV spatial alignment may represent an important secondary cue to aid selective integration of the correct inputs.…”

Section: Introductionmentioning

confidence: 80%

Spatial alignment between faces and voices improves selective attention to audio-visual speech

Fleming

Maddox

Shinn‐Cunningham

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The ability to see a talker's face has long been known to improve speech intelligibility in noise. This perceptual benefit depends on approximate temporal alignment between the auditory and visual speech components. However, the practical role that cross-modal spatial alignment plays in integrating audio-visual (AV) speech remains unresolved, particularly when competing talkers are present. In a series of online experiments, we investigated the importance of spatial alignment between corresponding faces and voices using a paradigm that featured both acoustic masking (speech-shaped noise) and attentional demands from a competing talker. Participants selectively attended a Target Talker's speech, then identified a word spoken by the Target Talker. In Exp. 1, we found improved task performance when the talkers' faces were visible, but only when corresponding faces and voices were presented in the same hemifield (spatially aligned). In Exp. 2, we tested for possible influences of eye position on this result. In auditory-only conditions, directing gaze toward the distractor voice reduced performance as predicted, but this effect could not fully explain the cost of AV spatial misalignment. Finally, in Exp. 3 and 4, we show that the effect of AV spatial alignment changes with noise level, but this was limited by a floor effect: due to the use of closed-set stimuli, participants were able to perform the task relatively well using lipreading alone. However, comparison between the results of Exp. 1 and Exp. 3 suggests that the cost of AV misalignment is larger at high noise levels. Overall, these results indicate that spatial alignment between corresponding faces and voices is important for AV speech integration in attentionally demanding communication settings.

show abstract

“…The reentry theory describes the bidirectional exchange of signals along reciprocal axonal fibers linking two or more brain areas, that are temporally correlated, and can educate each other. The human brain has a cortex to process the audio and visual stimuli together [39]. In a cocktail party, visual cues are not corrupted by background noise, reverberation, and interference speech thus providing a robust attractor for auditory attention [40].…”

Section: Introductionmentioning

confidence: 99%

Selective Listening by Synchronizing Speech With Lips

Pan

Tao

et al. 2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

A speaker extraction algorithm seeks to extract the speech of a target speaker from a multi-talker speech mixture when given a cue that represents the target speaker, such as a pre-enrolled speech utterance, or an accompanying video track. Visual cues are particularly useful when a pre-enrolled speech is not available. In this work, we don't rely on the target speaker's pre-enrolled speech, but rather use the target speaker's face track as the speaker cue, that is referred to as the auxiliary reference, to form an attractor towards the target speaker. We advocate that the temporal synchronization between the speech and its accompanying lip movements is a direct and dominant audio-visual cue. Therefore, we propose a self-supervised pretraining strategy, to exploit the speech-lip synchronization cue for target speaker extraction, which allows us to leverage abundant unlabeled in-domain data. We transfer the knowledge from the pre-trained model to the attractor encoder of the speaker extraction network. We show that the proposed speaker extraction network outperforms various competitive baselines in terms of signal quality, perceptual quality, and intelligibility, achieving state-of-the-art performance.

show abstract

Audio-visual spatial alignment improves integration in the presence of a competing audio-visual stimulus

Cited by 18 publications

References 81 publications

Towards understanding how we pay attention in naturalistic visual search settings

Towards understanding how we pay attention in naturalistic visual search settings

Spatial alignment between faces and voices improves selective attention to audio-visual speech

Selective Listening by Synchronizing Speech With Lips

Contact Info

Product

Resources

About