Speech Fine Structure Contains Critical Temporal Cues to Support Speech Segmentation

Abstract

1Segmenting the continuous speech stream into units for further perceptual and linguistic 2 analyses is fundamental to speech recognition. The speech amplitude envelope (SE) has 3 long been considered a fundamental temporal cue for segmenting speech. Does the 4 temporal fine structure (TFS), a significant part of speech signals often considered to 5 contain primarily spectral information, contribute to speech segmentation? Using 6 magnetoencephalography, we show that the TFS entrains cortical oscillatory responses 7 between 3-6 Hz and demonstrate, using mutual information analysis, that (i) the temporal 8 information in the TFS can be reconstructed from a measure of frame-to-frame spectral 9 change and correlates with the SE and (ii) that spectral resolution is key to the extraction 10 of such temporal information. Furthermore, we show behavioural evidence that, when the 11 SE is temporally distorted, the TFS provides cues for speech segmentation and aids 12 speech recognition significantly. Our findings show that it is insufficient to investigate 13 solely the SE to understand temporal speech segmentation, as the SE and the TFS derived 14 from a band-filtering method convey comparable, if not inseparable, temporal 15 information. We argue for a more synthetic view of speech segmentationthe auditory 16 system groups speech signals coherently in both temporal and spectral domains. 17 18

Modulation Spectra Capture EEG Responses to Speech Signals and Drive Distinct Temporal Response Functions

“…Our previous work (Teng, Tian, Doelling, & Poeppel, 2018b) showed that the onset responses were modulated by different frequency modulation spectra even though the same ramping windows were added to the stimuli of different 1/f modulation spectra. This interesting difference between the current experiment and Teng et al, 2018 suggests that not only the shape of amplitude envelopes but also spectral details of sounds significantly modulate auditory evoked responses (Oganian & Chang, 2018;Teng, Cogan, & Poeppel, 2018a).…”

Delta/theta band EEG differentially tracks low and high frequency speech-derived envelopes

“…The representation of speech in the brain is often examined by measuring the alignment of rhythmic brain activity to the acoustic envelope of the signal ( Ahissar et al., 2001 ; Ding et al., 2014 ; Giraud and Poeppel, 2012 ; Gross et al., 2013 ; Kayser et al., 2015 ; Oganian and Chang, 2019 ; Teng et al., 2019 ; Teoh et al., 2019 ). To quantify this alignment, many studies rely on the overall or broadband acoustic envelope, which describes the amplitude fluctuation of the signal across the full spectral range and which provides a convenient and low-dimensional representation for data analysis.…”

Robust EEG-Based Decoding of Auditory Attention With High-RMS-Level Speech Segments in Noisy Conditions

“…A higher reconstruction accuracy between EEG responses and the speech envelopes was found in the delta and theta bands than that in the higher frequency bands, which was consistent with the literature (e.g., Ding and Simon, 2013 ; Di Liberto et al, 2015 ). However, speech features in the time and spectral domain could all affect the speech perception and corresponding cortical responses (e.g., Biesmans et al, 2016 ; Teng et al, 2019 ). Future studies could systematically analyze how cortical responses track the speech features at different auditory-inspired narrow bands to better simulate the processing in the auditory peripheral and central systems.…”