Speech Production in Intracranial Electroencephalography: iBIDS Dataset

Verwoert, Maxime; Ottenhoff, Maarten C; Colon, Albert; Wagner, Louis; Tousseyn, Simon; Dijk, Hans van; Kubben, Pieter

doi:10.1101/2022.04.05.487183

Cited by 1 publication

(1 citation statement)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, a linear output layer was implemented for simplicity and comparability; however, more complex decoder paradigms, such as a GPT transformer stack may be better suited to more complex downstream tasks. The recent and growing corpus of publicly available data sets [49] can be leveraged to pool data from participants across experiments, and potentially across sensing paradigms, as long as the dataset includes electrode coordinates for the positional embedding.…”

Section: Discussionmentioning

confidence: 99%

Self-Supervised Learning of Neural Speech Representations From Unlabeled Intracranial Signals

et al. 2022

View full text Add to dashboard Cite

Neuroprosthetics have demonstrated the potential to decode speech from intracranial brain signals, and hold promise for one day returning the ability to speak to those who have lost it. However, data in this domain is scarce, highly variable, and costly to label for supervised modeling. In order to address these constraints, we present brain2vec, a transformer-based approach for learning feature representations from intracranial electroencephalogram data. Brain2vec combines a self-supervised learning methodology, neuroanatomical positional embeddings, and the contextual representations of transformers to achieve three novelties: (1) learning from unlabeled intracranial brain signals, (2) learning from multiple participants simultaneously, all while (3) utilizing only raw unprocessed data. To assess our approach, we use a leave-one-participant-out validation procedure to separate brain2vec's feature learning from the holdout participant's speech-related supervised classification tasks. With only two linear layers, we achieve 90% accuracy on a canonical speech detection task, 42% accuracy on a more challenging 4-class speech-related behavior recognition, and 53% accuracy when applied to a 10-class, few-shot word classification task. Combined with visualizations of unsupervised class separation in the learned features, our results evidence brain2vec's ability to learn highly generalized representations of neural activity without the need for labels or consistent sensor location.

show abstract

Section: Discussionmentioning

confidence: 99%