Speech intelligibility changes the temporal evolution of neural speech tracking

Chen, Ya-Ping; Schmidt, Fabian; Keitel, Anne; Roesch, Sebastian; Hauswald, Anne; Weisz, Nathan

doi:10.1016/j.neuroimage.2023.119894

Cited by 14 publications

(21 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Building on these results, we used Bayesian multilevel modeling to establish a link between neural speech tracking and behavior (Figure 3). Higher acoustic neural tracking is related to higher task performance, a finding also reported in a study that used vocoded speech (Chen et al, 2023). We also show that higher acoustic neural tracking is related to lower difficulty ratings.…”

Section: Predicting Behavior With Neural Trackingsupporting

confidence: 88%

“…Neural speech tracking has been proposed as an objective measure for speech intelligibility (Schmitt et al, 2022;Vanthornhout et al, 2018) along with a whole range of auditory and linguistic processes (Gillis et al, 2022). Previously, acoustic neural speech tracking has been related to behavioral measures such as speech intelligibility (Chen et al, 2023;Ding & Simon, 2013).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Reisinger

Gillis

Suess

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing of audiovisual stimuli. However, the interindividual variability of this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses from normal-hearing participants listening to audiovisual speech with or without an additional distractor speaker. Using temporal response functions (TRFs), we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After conducting a crucial control for speech acoustics, we show that lip movements effectively contribute to higher neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual benefit varied greatly among participants. Probing the behavioral relevance, we show that individuals who benefit more from lip movement information in terms of neural speech tracking, show a stronger drop in performance and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the benefit of lip movements in terms of neural speech tracking varies among individuals. Furthermore, we reveal its behavioral relevance by demonstrating negative consequences for behavior when visual speech is absent. Our results also offer potential implications for future objective assessments of audiovisual speech perception.

show abstract

Section: Predicting Behavior With Neural Trackingsupporting

confidence: 88%

Section: Introductionmentioning

confidence: 99%

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Reisinger

Gillis

Suess

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In analyzing the time-course of the speech-tracking response (TRF), we found that noise primarily impacted the mid-latency positive peak response around ∼150ms, but not the earlier negative peak at 100ms which is associated with low-level acoustic encoding (Salmelin, 2007). This is consistent with several recent studies that report correlations between mid-latency TRF components and intelligibility, particularly under noisy conditions (Broderick et al, 2018; Chen et al, 2023; Verschueren et al, 2020). Other studies reported modulation of mid-latency TRF components for degraded vs. clear speech (Kraus et al, 2021; Strauß et al, 2013), for background vs. target speech (Kaufman & Zion Golumbic, 2023; Orf et al, 2023) and for speech in an unfamiliar vs. familiar voice (Har-shai Yahav et al, 2023).…”

Section: Discussionsupporting

confidence: 93%

An ecological investigation of the effect of background noise on speech processing in a Virtual Classroom

Levy,

Korisky,

Zvilichovsky

et al. 2024

Preprint

View full text Add to dashboard Cite

Many real-life situations can be extremely noisy. Psychoacoustic studies have shown that background noise can have a detrimental effect on the ability to process and understand speech. However, most studies use stimuli and task designs that are highly artificial, limiting their generalization to more realistic contexts. Moreover, to date, we do not fully understand the neurophysiological consequences of trying to pay attention to speech in a noisy place.To address this lab to real-life gap and increase the ecological validity of speech in noise research, here we introduce a novel audiovisual Virtual Reality (VR) experimental platform. Combined with neurophysiological measurements of neural activity (EEG), eye-gaze and skin conductance (GSR) we studied the effects of background noise in a realistic context where the ability to process and understand continuous speech is especially important: A VR Classroom.Participants (n=32) sat in a VR Classroom and were told to pay attention to mini-lecture segments by a virtual teacher. Trials were either Quiet or contained background construction noise, emitted from outside the classroom window, which was either Continuous (drilling) or Intermittent (air hammers).Result show that background noise had a detrimental effect on learning outcomes, which was also accompanied by reduced neural tracking of the teacher’s speech. Comparison of the two noise types showed that the intermittent construction noise was more disruptive than continuous noise, as index by both behavioral and neural measures, and it also elicited higher skin-conductance levels, reflecting heightened arousal. Interesting, eye-gaze dynamics were not affected by the presence of noise.This study advances our understanding of the neurophysiological effects of background noise and extends it to more ecologically relevant contexts. It also emphasizes the role that temporal dynamics play for processing speech in noise, highlighting the need to consider the features of realistic noises, as we expand speech in noise research to increasingly realistic circumstances.

show abstract

“…A post hoc analysis of their stimuli revealed that vocoding decouples the syllable rate from the modulation rate of the speech signal. [106] also confirmed in their data that decreasing acoustic clarity was associated with a shift in the central frequency of tracking from the syllabic to acoustic modulation rate of speech. Hence, it is possible that, in our experiment, the frequency bandwidth of neural tracking may have shifted towards a faster rate in spectrally degraded conditions, although the overall magnitude of decoding accuracy was unchanged.…”

Section: Discussionmentioning

confidence: 73%

Neural decoding of the speech envelope: Effects of intelligibility and spectral degradation

MacIntyre,

Carlyon,

Goehring

2024

Preprint

View full text Add to dashboard Cite

During continuous speech perception, patterns of endogenous neural activity become time-locked to acoustic stimulus features, such as the speech amplitude envelope. This speech-brain coupling can be decoded using non-invasive brain imaging techniques, including electroencephalography (EEG). Methods like these may provide clinical use as an objective measure of stimulus encoding by the brain, for example, in the case of cochlear implant (CI) listening. Yet, the CI-transmitted speech signal is severely spectrally degraded, rendering its amenability to neural decoding unknown. Furthermore, interplay between acoustic and linguistic factors may lead to top-down modulation of perception, thereby challenging potential audiological applications. We assess neural decoding of the speech envelope under spectral degradation with EEG in acoustically hearing listeners (n = 38; 18-35 years old) using vocoded speech. Additionally, we dissociate sensory from higher-order processing by employing intelligible (English) and non-intelligible (Dutch) stimuli. Subject-specific and group decoders were trained to reconstruct the speech envelope from held-out EEG, with decoder significance determined via random permutation testing. Whereas speech envelope reconstruction did not vary by acoustic clarity, intelligible speech was associated with better decoding accuracy in general. Results were similar across subject-specific and group analyses, with less consistent effects of spectral degradation in group decoding. Permutation tests revealed possible differences in decoder statistical significance by experimental condition. In general, while robust neural decoding was observed at the group level, variability within participants would most likely prevent the clinical use of such a measure to differentiate levels of spectral degradation and intelligibility on an individual basis.

show abstract

Speech intelligibility changes the temporal evolution of neural speech tracking

Cited by 14 publications

References 67 publications

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

An ecological investigation of the effect of background noise on speech processing in a Virtual Classroom

Neural decoding of the speech envelope: Effects of intelligibility and spectral degradation

Contact Info

Product

Resources

About