Reconstructing Speech from Human Auditory Cortex

Abstract

Direct brain recordings from neurosurgical patients listening to speech reveal that the acoustic speech signals can be reconstructed from neural activity in auditory cortex.

Sound representation in higher language areas during language generation

“…3A). These findings are similar to those described in the superior temporal gyrus during listening (1,6).…”

“…This is true whether the encoded linguistic items will then be uttered or not (as when we think). Our findings concerning the correlation of cortical activity in high-level language areas to the sound envelope of the encoded linguistic items are symmetrical to those emerging by studying cortical activity during speech perception (3,5,6). Even if we assume that subvocalization with subthreshold activation of the phonatory apparatus is always active when we think (20), the correlation between ECoG and the envelope of the sound of the linguistic items involved is not limited to the late phonological and phonetic processing and the encoding of the articulatory motor commands that immediately precede sound production.…”

“…The correlation between the power envelope of the speech and the neural activity is maximal at low frequencies (2-7 Hz, theta range) corresponding to syllable rates, and becomes less precise at higher frequencies Hz, gamma range) corresponding to phoneme rates (1). Entrainment of neural activity to the speech envelope in auditory regions has allowed recognition of the phonetic features heard during speech perception (2)(3)(4)(5), and even the reconstruction of simple words (6). This evidence indicates that during listening, speech representation in the auditory cortex and adjacent areas of the superior temporal lobe reflects acoustic features directly related to linguistically defined phonological entities such as phonemes and syllables.…”

A Precluding But Not Ensuring Role of Entrained Low-Frequency Oscillations for Auditory Perception

“…Slow delta and theta oscillations entrain to sound sequences or speech, as evidenced by studies on humans (Luo and Poeppel, 2007;Howard and Poeppel, 2010;Ng et al, 2012;Pasley et al, 2012) or animals Kayser et al, 2009). One possibility by which entrainment can arise is that slow envelope modulations prominent in many natural sounds directly imprint on periodic excitability changes in cortical networks and effectively provide an intrinsic copy of the slow stimulus dynamics (Howard and Poeppel, 2010;Ding and Simon, 2012;Zion Golumbic et al, 2012).…”

A system identification approach to determining listening attention from EEG signals

“…Regression methods and stimulus reconstruction approaches have already been successfully applied to intracranial EEG or electroencephalographic (ECoG) data [4], [5], [6], MEG data [3], [7] and EEG data [8], [2]. Although impressive results can be obtained using ECoG data, ECoG measurements are invasive and can only be used with listeners under medical care; as such, these approaches are not plausible for everyday applications.…”