Dynamic time-locking mechanism in the cortical representation of spoken words

Abstract

Computational modeling of cortical responses to spoken words highlights the relevance of temporal tracking of spectrotemporal features, which is likely pivotal for transforming the acoustic-phonetic features into linguistic representations. AbstractHuman speech has a unique capacity to carry and communicate rich meanings. However, it is not known how the highly dynamic and variable perceptual signal is mapped to existing linguistic and semantic representations. In this novel approach, we utilized the natural acoustic variability of sounds and mapped them to magnetoencephalography (MEG) data using physiologically-inspired machine-learning models. We aimed at determining how well the models, differing in their representation of temporal information, serve to decode and reconstruct spoken words from MEG recordings in 16 healthy volunteers. We discovered that time-locking of the cortical activation to the unfolding speech input is crucial for the encoding of the acoustic-phonetic features. In contrast, time-locking was not highlighted in cortical processing of non-speech environmental sounds that conveyed the same meanings as the spoken words, including human-made sounds with temporal modulation content similar to speech. The amplitude envelope of the spoken words was particularly well reconstructed based on cortical evoked responses. Our results indicate that speech is encoded cortically with especially high temporal fidelity. This mechanism may contribute to the frequently reported entrainment of the cortical oscillations to the amplitude envelope of speech. Furthermore, the phoneme content was reflected in cortical evoked responses simultaneously with the spectrotemporal features, pointing to an instantaneous transformation of the unfolding acoustic features into linguistic representations during speech processing.

Selective auditory attention within naturalistic scenes modulates reactivity to speech sounds

“…In line with this interpretation, the magnitude of phase locking of auditory cortical responses to on‐going speech has been demonstrated to decrease with diminishing speech intelligibility (Peelle et al, 2013). Our present non‐invasive recordings are in line also with related electrocorticographic findings (Chang et al, 2010; Mesgarani & Chang, 2012; Zion Golumbic et al, 2013) and extend them by suggesting specific processing of speech sounds among other natural sounds, not only during early acoustic encoding (Nora et al, 2020) but also during real‐life‐like auditory processing, presumably via accentuated top‐down modulatory activity suggested by the current results.…”

“…We have recently shown that MEG responses to speech sounds reflect the sounds' spectrotemporal characteristics in a time‐locked manner, whereas similar time locking is not observed for environmental sounds or human non‐speech sounds with comparable spectrotemporal modulations (Nora et al, 2020). The results speak for neural activation specifically tracking the speech sound spectrogram, possibly essential for encoding relevant acoustic–phonetic features during speech processing.…”

“…Speech specificity is often studied by comparing brain responses with speech and acoustically well‐matched non‐speech stimuli (Overath et al, 2015; Peelle et al, 2013; Scott et al, 2000). With an alternative paradigm that made use of the natural acoustical variability within speech sounds and a machine‐learning approach, we recently demonstrated that magnetoencephalographic (MEG) auditory responses accurately follow the temporal unfolding of spoken words presented in isolation (Nora et al, 2020). Notably, a similar time‐locking mechanism was not evident for processing of environmental sounds, not even for non‐speech vocal sounds with spectrotemporal modulations comparable to spoken words and conveying the same meaning (e.g., coughing and laughter).…”

“…Notably, a similar time‐locking mechanism was not evident for processing of environmental sounds, not even for non‐speech vocal sounds with spectrotemporal modulations comparable to spoken words and conveying the same meaning (e.g., coughing and laughter). The result suggested that the acoustic‐phonetic content of speech sounds is tracked in a special manner compared with other natural sounds (Nora et al, 2020).…”

“…We predicted that, in accordance with earlier recordings (Chang et al, 2010; Hausfeld, Riecke, Valente, & Formisano, 2018; Mesgarani & Chang, 2012; Puvvada & Simon, 2017), the cortical activation pattern would primarily reflect the attended component of the complex auditory stimulus, especially during hierarchically later processing stages. Specifically, taken the notable time locking of auditory cortical responses to speech sounds (Nora et al, 2020), we hypothesized that attention to speech sounds, in particular, would modulate their cortical processing in an SER‐related manner, compared with other complex sounds especially in the left planum temporale (PT), which has been suggested to be particularly involved in analysis of spectrotemporally complex sounds (Griffiths & Warren, 2002) and showed sensitivity to speech SNR in our earlier study (Renvall et al, 2012).…”