Neural representation of linguistic feature hierarchy reflects second-language proficiency

Abstract

Acquiring a new language requires individuals to simultaneously and gradually learn linguistic attributes on multiple levels. Here, we investigated how this learning process changes the neural encoding of natural speech by assessing the encoding of the linguistic feature hierarchy in second-language listeners. Electroencephalography (EEG) signals were recorded from native Mandarin speakers with varied English proficiency and from native English speakers while they listened to audio-stories in English. We measured the temporal response functions (TRFs) for acoustic, phonemic, phonotactic, and semantic features in individual participants and found a main effect of proficiency on linguistic encoding. This effect of second-language proficiency was particularly prominent on the neural encoding of phonemes, showing stronger encoding of “new” phonemic contrasts (i.e., English contrasts that do not exist in Mandarin) with increasing proficiency. Overall, we found that the nonnative listeners with higher proficiency levels had a linguistic feature representation more similar to that of native listeners, which enabled the accurate decoding of language proficiency. This result advances our understanding of the cortical processing of linguistic information in second-language learners and provides an objective measure of language proficiency.

Envelope reconstruction of speech and music highlights stronger tracking of speech at low frequencies

“…The importance of low-frequency tracking (within the range generally considered "delta" in most studies) for speech is not new [14,29,53], but to our knowledge no study has suggested that low-frequency tracking is stronger for speech than other naturalistic stimuli. Recent studies also found that parietal weighting was increased for tracking phoneme and word surprisal [54] as well as semantic tracking of speech for native speakers but not non-native speakers [55]. The parietal weighting could be indicative of language-specific processing in the posterior temporal lobe [56] which was recently shown to be absent when listening to music [57], but because the activations are broad and without source localization it is difficult to identify the location definitively.…”

“…But we think that our observation of stronger low-frequency tracking (< 1 Hz) for speech is notable for auditory attention decoding work. Low-frequency tracking may relate to several cognitive aspects of speech processing such as semantics, prosody, surprise, attention, comprehension, and language proficiency [14,29,36,55,73,74]. If other naturalistic sounds are not sensitive to this frequency range, then there could be considerable benefit to focusing on this frequency range to identify the locus of attention of a talker and isolate the most relevant speech feature to which a user is engaged.…”

Accurate Decoding of Imagined and Heard Melodies

“…Recent research on speech perception provided substantial evidence indicating that low-frequency neural signals in the delta- and theta-bands encode both acoustic features, such as the sound envelope and higher-level linguistic properties ( Lalor and Foxe, 2010 ; Ding et al, 2014 ; Di Liberto et al, 2015 , 2021b ; Brodbeck et al, 2018 ; Alday, 2019 ; Obleser and Kayser, 2019 ). This multifaceted encoding has also been measured in the context of music, showing that non-invasive neural recordings reflect properties such as tonal structure ( Koelsch and Friederici, 2003 ; Koelsch and Siebel, 2005 ; Sankaran et al, 2018 ), beat ( Tal et al, 2017 ), and melodic expectations ( Omigie et al, 2013 ; Di Liberto et al, 2020 ).…”

“…The ability to measure this multifaceted neural encoding of melodies with non-invasive brain recordings provides us with new opportunities to unveil the neural encoding of melodies and its precise role in music perception. The encoding modelling framework constitutes an effective solution to study the neural processing of complex sounds, such as melodies, by teasing apart its various cortical contributors ( Koelsch, 2011 ; Brodbeck et al, 2018 ; Obleser and Kayser, 2019 ; Di Liberto et al, 2021b ). While that work informed us on which properties of music are encoded in cortical signals measured, hence contributing to our understanding of how the human brain processes melodies, the present study investigated the inverse question: can we use such cortical signals to identify the corresponding melodies that were either listened to or imagined?…”

Extracting Different Levels of Speech Information from EEG Using an LSTM-Based Model

“…1,2,3,4]. It has been shown that this approach can be used as an objective measure of how well speech is understood by a listener [3,4,5]. Real-time and accurate speech decoding from the brain has other potential applications such as Brain-Computer Interfaces (BCIs).…”