Causal inference and temporal predictions in audiovisual perception of speech and music

Noppeney, Uta; Lee, Hwee Ling

doi:10.1111/nyas.13615

“…When integrating audiovisual information, we rely on the congruence between auditory and visual cues in terms of their object identity, their spatial location, or their temporal synchrony (Noppeney and Lee 2018). To date, while previous studies have used functional magnetic resonance imaging (fMRI) to investigate the anatomical sites of audiovisual interactions (Gau and Noppeney 2016;Werner and Noppeney 2010), it is still unclear how congruent auditory input modulates the time course of visual object processing.…”

Section: Introductionmentioning

confidence: 99%

Auditory and Semantic Cues Facilitate Decoding of Visual Object Category in MEG

Brandman

¹

,

Avancini

²

,

Leticevscaia

³

et al. 2019

View full text Add to dashboard Cite

Sounds (e.g., barking) help us to visually identify objects (e.g., a dog) that are distant or ambiguous. While neuroimaging studies have revealed neuroanatomical sites of audiovisual interactions, little is known about the time-course by which sounds facilitate visual object processing. Here we used magnetoencephalography (MEG) to reveal the time-course of the facilitatory influence of natural sounds (e.g., barking) on visual object processing, and compared this to the facilitatory influence of spoken words (e.g., "dog"). Participants viewed images of blurred objects preceded by a task-irrelevant natural sound, a spoken word, or uninformative noise. A classifier was trained to discriminate multivariate sensor patterns evoked by animate and inanimate intact objects with no sounds, presented in a separate experiment, and tested on sensor patterns evoked by the blurred objects in the three auditory conditions. Results revealed that both sounds and words, relative to uninformative noise, significantly facilitated visual object category decoding between 300-500 ms after visual onset. We found no evidence for earlier facilitation by sounds than by words. These findings provide evidence for a semantic route of facilitation by both natural sounds and spoken words, whereby the auditory input first activates semantic object representations, which then modulate the visual processing of objects.

show abstract

“…When integrating audiovisual information, we rely on the congruence between auditory and visual cues in terms of their object identity, their spatial location, or their temporal synchrony (Noppeney and Lee 2018). To date, while previous studies have used functional magnetic resonance imaging (fMRI) to investigate the anatomical sites of audiovisual interactions (Gau and Noppeney 2016;Werner and Noppeney 2010), it is still unclear how congruent auditory input modulates the time course of visual object processing.…”

Section: Introductionmentioning

confidence: 99%

Auditory and semantic cues facilitate decoding of visual object category in MEG

Brandman

¹

,

Avancini

²

,

Leticevscaia

³

et al. 2019

Preprint

View full text Add to dashboard Cite

Sounds (e.g., barking) help us to visually identify objects (e.g., a dog) that are distant or ambiguous. While neuroimaging studies have revealed neuroanatomical sites of audiovisual interactions, little is known about the time-course by which sounds facilitate visual object processing. Here we used magnetoencephalography (MEG) to reveal the time-course of the facilitatory influence of natural sounds (e.g., barking) on visual object processing, and compared this to the facilitatory influence of spoken words (e.g., "dog"). Participants viewed images of blurred objects preceded by a task-irrelevant natural sound, a spoken word, or uninformative noise. A classifier was trained to discriminate multivariate sensor patterns evoked by animate and inanimate intact objects with no sounds, presented in a separate experiment, and tested on sensor patterns evoked by the blurred objects in the three auditory conditions. Results revealed that both sounds and words, relative to uninformative noise, significantly facilitated visual object category decoding between 300-500 ms after visual onset. We found no evidence for earlier facilitation by sounds than by words. These findings provide evidence for a semantic route of facilitation by both natural sounds and spoken words, whereby the auditory input first activates semantic object representations, which then modulate the visual processing of objects.

show abstract

“…Previous research showed subtle differences between musicians and non-musicians in AV perception (Musacchia et al, 2008; Lee and Noppeney, 2011; Paraskevopoulos et al, 2012; Proverbio et al, 2016). For example, a perceiver's prior expectations can influence temporal integration, and relevant training can generate more precise temporal predictions (Noppeney and Lee, 2018) leading to higher sensitivity to AV misalignments (for behavioral study see, Behne et al, 2013, for EEG study see, Behne et al, 2017). Another study (Petrini et al, 2009a) has shown that while musicians have a more refined integration window for AV music perception compared to non-musicians, they are also more accurate at predicting an upcoming sound when the visual information is missing in AV music perception (Petrini et al, 2009b).…”

Section: Discussionmentioning

confidence: 99%

Musical Expertise Affects Audiovisual Speech Perception: Findings From Event-Related Potentials and Inter-trial Phase Coherence

Sorati¹,

Behne²

2019

View full text Add to dashboard Cite

In audiovisual speech perception, visual information from a talker's face during mouth articulation is available before the onset of the corresponding audio speech, and thereby allows the perceiver to use visual information to predict the upcoming audio. This prediction from phonetically congruent visual information modulates audiovisual speech perception and leads to a decrease in N1 and P2 amplitudes and latencies compared to the perception of audio speech alone. Whether audiovisual experience, such as with musical training, influences this prediction is unclear, but if so, may explain some of the variations observed in previous research. The current study addresses whether audiovisual speech perception is affected by musical training, first assessing N1 and P2 event-related potentials (ERPs) and in addition, inter-trial phase coherence (ITPC). Musicians and non-musicians are presented the syllable, /ba/ in audio only (AO), video only (VO), and audiovisual (AV) conditions. With the predictory effect of mouth movement isolated from the AV speech (AV−VO), results showed that, compared to audio speech, both groups have a lower N1 latency and P2 amplitude and latency. Moreover, they also showed lower ITPCs in the delta, theta, and beta bands in audiovisual speech perception. However, musicians showed significant suppression of N1 amplitude and desynchronization in the alpha band in audiovisual speech, not present for non-musicians. Collectively, the current findings indicate that early sensory processing can be modified by musical experience, which in turn can explain some of the variations in previous AV speech perception research.

show abstract

Causal inference and temporal predictions in audiovisual perception of speech and music

Cited by 29 publications

References 100 publications

Auditory and Semantic Cues Facilitate Decoding of Visual Object Category in MEG

Auditory and Semantic Cues Facilitate Decoding of Visual Object Category in MEG

Auditory and semantic cues facilitate decoding of visual object category in MEG

Musical Expertise Affects Audiovisual Speech Perception: Findings From Event-Related Potentials and Inter-trial Phase Coherence

Contact Info

Product

Resources

About