Multi-Head Self-Attention Model for Classification of Temporal Lobe Epilepsy Subtypes

Abstract

As a long-standing chronic disease, Temporal Lobe Epilepsy (TLE), resulting from abnormal discharges of neurons and characterized by recurrent episodic central nervous system dysfunctions, has affected more than 70% of drug-resistant epilepsy patients across the world. As the etiology and clinical symptoms are complicated, differential diagnosis of TLE mainly relies on experienced clinicians, and specific diagnostic biomarkers remain unclear. Though great effort has been made regarding the genetics, pathology, and neuroimaging of TLE, an accurate and effective diagnosis of TLE, especially the TLE subtypes, remains an open problem. It is of a great importance to explore the brain network of TLE, since it can provide the basis for diagnoses and treatments of TLE. To this end, in this paper, we proposed a multi-head self-attention model (MSAM). By integrating the self-attention mechanism and multilayer perceptron method, the MSAM offers a promising tool to enhance the classification of TLE subtypes. In comparison with other approaches, including convolutional neural network (CNN), support vector machine (SVM), and random forest (RF), experimental results on our collected MEG dataset show that the MSAM achieves a supreme performance of 83.6% on accuracy, 90.9% on recall, 90.7% on precision, and 83.4% on F1-score, which outperforms its counterparts. Furthermore, effectiveness of varying head numbers of multi-head self-attention is assessed, which helps select the optimal number of multi-head. The self-attention aspect learns the weights of different signal locations which can effectively improve classification accuracy. In addition, the robustness of MSAM is extensively assessed with various ablation tests, which demonstrates the effectiveness and generalizability of the proposed approach.

CCheXR‐Attention: Clinical concept extraction and chest x‐ray reports classification using modified Mogrifier and bidirectional LSTM with multihead attention

“…We evaluated the performance of the CCheXR‐Att model in comparison with several baseline and state‐of‐the‐art models, including LSTM, BiLSTM, CNN‐BiLSTM‐CRF, 37 BoW‐based model, 38 FasTag, 39 BERT LARGE , 40 Hybrid BiGRU, 42 Fine‐Tuned BERT, 43 Mogrifier LSTM, 46 and MSAM 50 …”

“…Mogrifier LSTM 46 introduces a “Mogrifier” update, a gating mechanism that enhances LSTM networks by integrating information from different time steps, improving their ability to capture complex dependencies in sequential data. MSAM 50 employs self‐attention mechanisms with MHA heads to capture temporal dependencies and patterns effectively. This allows the model to focus on the most relevant features, enabling the accurate classification of radiological concepts.…”

“…We evaluated the performance of the CCheXR-Att model in comparison with several baseline and state-of-the-art models, including LSTM, BiLSTM, CNN-BiLSTM-CRF, 37 BoW-based model, 38 FasTag, 39 BERT LARGE , 40 Hybrid BiGRU, 42 Fine-Tuned BERT, 43 Mogrifier LSTM, 46 and MSAM. 50 LSTM relies on a single LSTM network to represent the sentence. The final sentence representation is derived by averaging all hidden states.…”

CircRNA‐associated ceRNA regulatory networks as emerging mechanisms governing the development and biophysiopathology of epilepsy

“…7 Annually, there are 67.77 new cases of epilepsy per one hundred thousand people worldwide. 8 Epilepsy, characterized by recurrent and unpredictable seizures, arises from episodic brain dysfunction. Partial seizures manifest initially in a specific brain region, whereas generalized seizures involve both hemispheres from their onset.…”

“…The global prevalence of epilepsy ranges from 6.38% to 7.60% 7 . Annually, there are 67.77 new cases of epilepsy per one hundred thousand people worldwide 8 . Epilepsy, characterized by recurrent and unpredictable seizures, arises from episodic brain dysfunction.…”