Neural representation of linguistic feature hierarchy reflects second-language proficiency

Abstract

Word count (excluding abstract, title page, references and methods): 6094. Acknowledgements:The authors would like to thank Michael Broderick for his help with the semantic dissimilarity analysis. The authors also thank Adam Soussana and Ghislain de Labbey for their help with a pilot version of this experiment. AbstractAcquiring a new language requires a simultaneous and gradual learning of multiple levels of linguistic attributes. Here, we investigated how this process changes the neural encoding of natural speech by assessing the encoding of the linguistic feature hierarchy in second-language listeners. Electroencephalography (EEG) signals were recorded during English story listening from native Mandarin speakers with varied English proficiency and from native English speakers. The neural encoding of acoustic, phonemic, phonotactic, and semantic features was measured in individual participants. We found that linguistic feature representation in nonnative listeners progressively converged to that of native listeners with proficiency, which enabled accurate decoding of language proficiency. This effect of second-language proficiency was particularly prominent on the neural encoding of phonemes, showing stronger encoding of "new" phonemic contrasts (i.e. English contrasts that do not exist in Mandarin) with increasing proficiency. This detailed view advances our understanding of the cortical processing of linguistic information in second-language learners and provides an objective measure of language proficiency. be affected differently by proficiency, with some of them becoming more native-like than others for proficient L2 users. Part of the evidence comes from electro-and magneto-encephalography (EEG and MEG respectively) research, which showed the effect of proficiency at the levels of phonemes 13 , syntax 14,15 , and semantics 16 . These studies measured the changes in well-known event-related potentials, such as the MMN, N400, and P600. These approaches, however, use unnatural speech stimuli (e.g., isolated syllables or violative speech sentences) which do not fully and realistically activate the specialized speech cortex [17][18][19] . In addition, these approaches consider various levels of speech perception independently and in isolation. Language learning, on the other hand, involves simultaneous acquisition of novel phonetic contrasts 20,21 , new syllabic structures (phonotactics) 22 , and new words. A more complete view of the neural basis of language learning therefore requires a joint study of multiple levels of the linguistic hierarchy to advance our understanding of L2 perception by informing us on the precise effect of proficiency on the cortical processing strategies that underpin sound and language perception 23-25 .Previous effort in using naturalistic speech stimuli to study language proficiency showed a modulation of EEG phase-synchronization during naturalistic speech listening both at sub-cortical (FFR 26,27 ) and cortical levels (gamma EEG synchrony 28,29 ). Specifically, stronger synchron...

Heard or understood? Neural tracking of language features in a comprehensible story, an incomprehensible story and a word list

“…Overall our results regarding the observed patterns in the TRFs to language features converge with the findings of Di Liberto et al (2021) and Broderick et al (2020): the worse the listener can integrate a word in its context due to lower language proficiency or higher word scrambling, the less prominent the N400 response. Therefore, activation of a word and integration of the word in its context are more difficult, explaining why no significant clusters are found for both word surprisal and word frequency for the Frisian story.…”

“…Overall our results regarding the observed patterns in the TRFs to language features converge with the findings of Di Liberto et al (2021) and Broderick et al (2020): the worse the listener can integrate a word in its context due to lower language proficiency or higher word scrambling, the less prominent the N400 19 preprint (which was not certified by peer review) is the author/funder. All rights reserved.…”

“…More recently, it was assessed whether language tracking is related to behavioral speech comprehension. By comparing listeners listening to their second language to native speakers of the language, Di Liberto et al (2021) showed that as language proficiency increases, the response around 400 ms becomes more similar to a native speaker. Two other studies manipulated the amount of speech comprehension by manipulating the speech via word scrambling (Broderick et al, 2020) or manipulating the speech rate (Verschueren et al, 2022).…”

Envelope reconstruction of speech and music highlights stronger tracking of speech at low frequencies

“…The importance of low-frequency tracking (within the range generally considered “delta” in most studies) for speech is not new [ 14 , 29 , 53 ], but to our knowledge no study has suggested that low-frequency tracking is stronger for speech than other naturalistic stimuli. Recent studies also found that parietal weighting was increased for tracking phoneme and word surprisal [ 54 ] as well as semantic tracking of speech for native speakers but not non-native speakers [ 55 ]. The parietal weighting could be indicative of language-specific processing in the posterior temporal lobe [ 56 ] which was recently shown to be absent when listening to music [ 57 ], but because the activations are broad and without source localization it is difficult to identify the location definitively.…”

“…But we think that our observation of stronger low-frequency tracking (< 1 Hz) for speech is notable for auditory attention decoding work. Low-frequency tracking may relate to several cognitive aspects of speech processing such as semantics, prosody, surprise, attention, comprehension, and language proficiency [ 14 , 29 , 36 , 55 , 73 , 74 ]. If other naturalistic sounds are not sensitive to this frequency range, then there could be considerable benefit to focusing on this frequency range to identify the locus of attention of a talker and isolate the most relevant speech feature to which a user is engaged.…”