Gated Transformer for Decoding Human Brain EEG Signals

Tao, Yunzhe; Sun, Tao; Muhamed, Aashiq; Genç, Şahika; Jackson, Dylan; Arsanjani, Ali; Yaddanapudi, Suri; Li, Liang; Kumar, Pardeep

doi:10.1109/embc46164.2021.9630210

Cited by 26 publications

(26 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While some studies showed that the Transformer model outperformed LSTM models in ASR 272829 , the studies on EEG signals have also successfully applied Transformer models for BCI 30313233343536 . Notably, studies by Tao et al 31 and Siddhad et al 32 showed that Transformer models outperformed LSTM models, even when they were trained on a small dataset, that is, of less than 10h. This dataset size is considerably smaller than that typically employed in ASR, which usually comprises over 1,000h of speech.…”

Section: Discussionmentioning

confidence: 99%

Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech

Komeiji,

Mitsuhashi,

Iimura

et al. 2024

Preprint

View full text Add to dashboard Cite

Several attempts for speech brain–computer interfacing (BCI) have been made to decode phonemes, sub-words, words, or sentences using invasive measurements, such as the electrocorticogram (ECoG), during auditory speech perception, overt speech, or imagined (covert) speech. Decoding sentences from covert speech is a challenging task. Sixteen epilepsy patients with intracranially implanted electrodes participated in this study, and ECoGs were recorded during overt speech, covert speech, and passive listening of eight Japanese sentences, each consisting of three tokens. A Transformer neural network model was applied to decode text sentences from covert speech, which was trained using ECoGs obtained during overt speech. We first examined the proposed Transformer model using the same task for training and testing and then evaluated the model’s performance when trained with overt or perception tasks for decoding covert speech. The Transformer model trained on covert speech achieved an average token error rate (TER) of 46.6% for decoding covert speech, whereas the model trained on overt speech achieved a comparable TER of 46.3% (p >0.05;d= 0.07). Therefore the challenge of collecting training data for covert speech can be addressed using overt speech. The performance of covert speech can improve by using large amounts of overt speech.

show abstract

Section: Discussionmentioning

confidence: 99%

Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech

Komeiji,

Mitsuhashi,

Iimura

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Vision Transformer (ViT) and its variants are highly capable of capturing global contextual information and longrange dependence in attention-based modeling of a plethora of modalities, including EEG spectra (Bagchi and Bathula 2022;Tao et al 2021), but are not capable enough to learn high-frequency components that are crucial for fine-grained information extraction and classification (Wang et al 2022;Bai et al 2022). Recent years have witness significant advance in examination of ViT from spectral domain, which contributes to resolving important gaps such as attention collapse (Wang et al 2022).…”

Section: Vision Transformermentioning

confidence: 99%

“…Seizure dynamics is characterized by high-frequency outbursts of epileptiform abnormalities. Recent advances in attention-based deep neural networks, especially Transformers, have prompted augmented representational learning for EEG classification tasks (Bagchi and Bathula 2022;Siddhad et al 2022;Tao et al 2021;Sun, Xie, and Zhou 2021). However, the problem of loss of relevant clinical biomarkers of irregularly altered EEG signals has not been explicitly addressed.…”

Section: Introductionmentioning

confidence: 99%

The Shanghai alliance of multilingual researchers: Fudan University, Tongji University, Shanghai University of Finance and Economics, and Shanghai International Studies University, China

et al. 2022

View full text Add to dashboard Cite

Patient-independent detection of epileptic activities based on visual spectral representation of continuous EEG (cEEG) has been widely used for diagnosing epilepsy. However, precise detection remains a considerable challenge due to subtle variabilities across subjects, channels and time points. Thus, capturing fine-grained, discriminative features of EEG patterns, which is associated with high-frequency textural information, is yet to be resolved. In this work, we propose Scattering Transformer (ScatterFormer), an invariant scattering transform-based hierarchical Transformer that specifically pays attention to subtle features. In particular, the disentangled frequency-aware attention (FAA) enables the Transformer to capture clinically informative high-frequency components, offering a novel clinical explainability based on visual encoding of multichannel EEG signals. Evaluations on two distinct tasks of epileptiform detection demonstrate the effectiveness our method. Our proposed model achieves median AUCROC and accuracy of 98.14%, 96.39% in patients with Rolandic epilepsy. On a neonatal seizure detection benchmark, it outperforms the state-of-the-art by 9% in terms of average AUCROC.

show abstract

“…They divided EEG signals into different brain regions according to Power Spectral Density (PSD) [206] and realized the hierarchical learning of spatial information from the electrode level to the brain region level using parallel selfattention. Tao et al [202] established a Gated Recurrent Unit Transformer (GRUGate Transformer) to acquire the longterm dependence of EEG signals. The gating mechanism stabilized the GRUGate training process and was assessed in EEG datasets for human brain-visual and motor imagery.…”

Section: Eeg Processingmentioning

confidence: 99%

Understanding the brain with attention: A survey of transformers in brain sciences

Chen,

Wang,

Chen

et al. 2023

Brain-X

View full text Add to dashboard Cite

Owing to their superior capabilities and advanced achievements, Transformers have gradually attracted attention with regard to understanding complex brain processing mechanisms. This study aims to comprehensively review and discuss the applications of Transformers in brain sciences. First, we present a brief introduction of the critical architecture of Transformers. Then, we overview and analyze their most relevant applications in brain sciences, including brain disease diagnosis, brain age prediction, brain anomaly detection, semantic segmentation, multi‐modal registration, functional Magnetic Resonance Imaging (fMRI) modeling, Electroencephalogram (EEG) processing, and multi‐task collaboration. We organize the model details and open sources for reference and replication. In addition, we discuss the quantitative assessments, model complexity, and optimization of Transformers, which are topics of great concern in the field. Finally, we explore possible future challenges and opportunities, exploiting some concrete and recent cases to provoke discussion and innovation. We hope that this review will stimulate interest in further research on Transformers in the context of brain sciences.

show abstract

Gated Transformer for Decoding Human Brain EEG Signals

Cited by 26 publications

References 16 publications

Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech

Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech

The Shanghai alliance of multilingual researchers: Fudan University, Tongji University, Shanghai University of Finance and Economics, and Shanghai International Studies University, China

Understanding the brain with attention: A survey of transformers in brain sciences

Contact Info

Product

Resources

About