Imagined speech can be decoded from low- and cross-frequency features in perceptual space

Proix, Timothée; Jd, Saa; Christen, Andy; Martin, Stéphanie; Bn, Pasley; Rt, Knight; Tian, Xing; Poeppel, David; Wk, Doyle; Devinsky, Orrin; Arnal, Luc H.; Mégevand, Pierre; Giraud, Anne-Lise

doi:10.1101/2021.01.26.428315

Cited by 6 publications

(7 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings suggests that motor-auditory interaction is essential in hierarchical meter imagery, which may apply to meaningful nested structure of imagined speech as well (Tian and Poeppel, 2012;Proix et al, 2021). We analyzed connectivity at the beat rate, assuming that the beat rate is a reasonable proxy for the importance of motor to auditory flow, since it turned out that we did not have sufficient data to robustly fit an MVAR model capable of capturing causal flow at the meter frequencies.…”

Section: Neural Substrates Of Meter Imaginationmentioning

confidence: 99%

How Do You Feel the Rhythm: Dynamic Motor-Auditory Interactions Are Involved in the Imagination of Hierarchical Timing

2021

View full text Add to dashboard Cite

Predicting and organizing patterns of events is important for humans to survive in a dynamically changing world. The motor system has been proposed to be actively, and necessarily, engaged in not only the production but the perception of rhythm by organizing hierarchical timing that influences auditory responses. It is not yet well understood how the motor system interacts with the auditory system to perceive and maintain hierarchical structure in time. This study investigated the dynamic interaction between auditory and motor functional sources during the perception and imagination of musical meters. We pursued this using a novel method combining high-density EEG, EMG and motion capture with independent component analysis to separate motor and auditory activity during meter imagery while robustly controlling against covert movement. We demonstrated that endogenous brain activity in both auditory and motor functional sources reflects the imagination of binary and ternary meters in the absence of corresponding acoustic cues or overt movement at the meter rate. We found clear evidence for hypothesized motor-to-auditory information flow at the beat rate in all conditions, suggesting a role for top-down influence of the motor system on auditory processing of beat-based rhythms, and reflecting an auditory-motor system with tight reciprocal informational coupling. These findings align with and further extend a set of motor hypotheses from beat perception to hierarchical meter imagination, adding supporting evidence to active engagement of the motor system in auditory processing, which may more broadly speak to the neural mechanisms of temporal processing in other human cognitive functions.

show abstract

Section: Neural Substrates Of Meter Imaginationmentioning

confidence: 99%

How Do You Feel the Rhythm: Dynamic Motor-Auditory Interactions Are Involved in the Imagination of Hierarchical Timing

2021

View full text Add to dashboard Cite

show abstract

“…Here we found that low-gamma activity was the only feature that specifically predicted language expression in TD children, a logical finding as language expression is tightly related to the transformation of phonetic into articulatory features at the same timescale [87][88][89][90][91].…”

Section: Low-gamma Power Predicts Language Expression In Td But Not Children With Asdmentioning

confidence: 50%

Neural oscillation coupling selectively predicts speech reception in young children with Autism Spectrum Disorder

Wang

Delgado²,

Marchesotti

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Communication difficulties in autism spectrum disorder (ASD) involve a speech reception deficit, whose biological causes are not yet identified. This deficit could denote atypical neuronal ensemble activity, as reflected by neural oscillations. Atypical cross-frequency oscillation coupling in particular could disrupt the possibility to jointly track and predict dynamic acoustic stimuli, a dual process that is essential for speech comprehension. Whether such oscillation anomalies can already be found in very young children with ASD, and with what specificity they relate to individual language reception capacity is unknown. In this study, neural activity was collected using EEG in 64 very young children with and without ASD (mean age 3) while they were exposed to naturalistic-continuous speech via an age-appropriate cartoon. EEG power typically associated with phrase-level chunking (delta, 1-3Hz), phonemic encoding (low-gamma, 25-35Hz) and top-down control (beta, 12-20Hz) was markedly reduced in ASD relative to typically developing (TD) children. Speech neural-tracking by delta and theta oscillations was also weaker in ASD than TD children. Critically, children with ASD exhibited slightly atypical theta/gamma coupling (PAC) involving a higher-than-normal gamma frequency, and markedly atypical beta/gamma PAC. Even though many oscillation features were atypical in our sample of 31 very young children with ASD, the beta/gamma coupling anomaly was the single best predictor of individual speech reception difficulties. These findings suggest that early interventions targeting the normalization of low-gamma and low-beta activity, might help young children with ASD to engage more in oral interactions.

show abstract

“…Performance on the Speech-related Behavior Recognition task, while comparatively exhibiting the weakest performance, can also be considered the most challenging of the three classification tasks. The neural circuits for perceiving speech, and producing overt, mouthed, and imagined speech, are highly intertwined [43], [46], [47]. Nevertheless, it is encouraging that the context representations of the model appear to encode some neural correlates of these behaviors.…”

Section: Discussionmentioning

confidence: 99%

Self-Supervised Learning of Neural Speech Representations From Unlabeled Intracranial Signals

et al. 2022

View full text Add to dashboard Cite

Neuroprosthetics have demonstrated the potential to decode speech from intracranial brain signals, and hold promise for one day returning the ability to speak to those who have lost it. However, data in this domain is scarce, highly variable, and costly to label for supervised modeling. In order to address these constraints, we present brain2vec, a transformer-based approach for learning feature representations from intracranial electroencephalogram data. Brain2vec combines a self-supervised learning methodology, neuroanatomical positional embeddings, and the contextual representations of transformers to achieve three novelties: (1) learning from unlabeled intracranial brain signals, (2) learning from multiple participants simultaneously, all while (3) utilizing only raw unprocessed data. To assess our approach, we use a leave-one-participant-out validation procedure to separate brain2vec's feature learning from the holdout participant's speech-related supervised classification tasks. With only two linear layers, we achieve 90% accuracy on a canonical speech detection task, 42% accuracy on a more challenging 4-class speech-related behavior recognition, and 53% accuracy when applied to a 10-class, few-shot word classification task. Combined with visualizations of unsupervised class separation in the learned features, our results evidence brain2vec's ability to learn highly generalized representations of neural activity without the need for labels or consistent sensor location.

show abstract

Imagined speech can be decoded from low- and cross-frequency features in perceptual space

Cited by 6 publications

References 63 publications

How Do You Feel the Rhythm: Dynamic Motor-Auditory Interactions Are Involved in the Imagination of Hierarchical Timing

How Do You Feel the Rhythm: Dynamic Motor-Auditory Interactions Are Involved in the Imagination of Hierarchical Timing

Neural oscillation coupling selectively predicts speech reception in young children with Autism Spectrum Disorder

Self-Supervised Learning of Neural Speech Representations From Unlabeled Intracranial Signals

Contact Info

Product

Resources

About