fNIRS Based Multi-Class Mental Workload Classification Using Recurrence Plots and CNN-LSTM

Mughal, Nabeeha Ehsan; Khalil, Khurram; Khan, Muhammad Jawad

doi:10.1109/aims52415.2021.9466084

Cited by 8 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, BCI is a neurofeedback method that can enhance the quality of life of patients suffering from serious motor debilities due to tetraplegia (Benaroch et al, 2021 ), stroke (Mane et al, 2020 ), and other spinal cord injuries (Al-Taleb et al, 2019 ). BCI also has applications in neurorehabilitation, communication and control, motor therapy and recovery, brain monitoring, and neuro-ergonomics (Asgher et al, 2020a , b ; Mughal et al, 2021 ). The BCI analyzes a biosignal measured from a healthy subject to predict some intangible aspects of their cognitive state.…”

Section: Introductionmentioning

confidence: 99%

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

et al. 2022

Self Cite

View full text Add to dashboard Cite

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

show abstract

Section: Introductionmentioning

confidence: 99%

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…This type of hybrid architecture has shown better performance than many other state-of-the-art models in many time series applications. In recent neuroimaging studies, this hybrid architecture has found applications such as, EEG data for screening of depression with an accuracy of 99% (Sharma et al, 2021 ), mental state monitoring (workload, vigilance, fatigue) using fNIRS with an accuracy of 85.9% (Mughal et al, 2021 ), or epileptic seizure recognition using EEG with an accuracy of 99.39% (Xu et al, 2020 ). In these applications, the CNN architectures serve the dual purpose of filtering out noise from the input data and extracting valuable information crucial for the final prediction; While LSTM architectures possess the capability to identify both short-term and long-term dependencies.…”

Section: Discussionmentioning

confidence: 99%

Empirical comparison of deep learning models for fNIRS pain decoding

Fernandez Rojas,

Joseph,

Bargshady

et al. 2024

Front. Neuroinform.

View full text Add to dashboard Cite

IntroductionPain assessment is extremely important in patients unable to communicate and it is often done by clinical judgement. However, assessing pain using observable indicators can be challenging for clinicians due to the subjective perceptions, individual differences in pain expression, and potential confounding factors. Therefore, the need for an objective pain assessment method that can assist medical practitioners. Functional near-infrared spectroscopy (fNIRS) has shown promising results to assess the neural function in response of nociception and pain. Previous studies have explored the use of machine learning with hand-crafted features in the assessment of pain.MethodsIn this study, we aim to expand previous studies by exploring the use of deep learning models Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and (CNN-LSTM) to automatically extract features from fNIRS data and by comparing these with classical machine learning models using hand-crafted features.ResultsThe results showed that the deep learning models exhibited favourable results in the identification of different types of pain in our experiment using only fNIRS input data. The combination of CNN and LSTM in a hybrid model (CNN-LSTM) exhibited the highest performance (accuracy = 91.2%) in our problem setting. Statistical analysis using one-way ANOVA with Tukey's (post-hoc) test performed on accuracies showed that the deep learning models significantly improved accuracy performance as compared to the baseline models.DiscussionOverall, deep learning models showed their potential to learn features automatically without relying on manually-extracted features and the CNN-LSTM model could be used as a possible method of assessment of pain in non-verbal patients. Future research is needed to evaluate the generalisation of this method of pain assessment on independent populations and in real-life scenarios.

show abstract

“…Additionally, Wang et al [13] proposed incorporating delayed hemodynamic response as domain knowledge into fNIRS classification to enhance classification accuracy. Examining the results from the last epoch of each cross-validation, and observing nearly similar accuracy and confidence levels between (a) [14] and (b) [13]. However, the calibration of (b) surpasses that of (a).…”

Section: Introductionmentioning

confidence: 89%

“…Fig.1. Examining the results from the last epoch of each cross-validation, and observing nearly similar accuracy and confidence levels between (a)[14] and (b)[13]. However, the calibration of (b) surpasses that of (a).…”

mentioning

confidence: 89%

“…1D-CNN [9], CNN [8], LSTM [8], CNN+LSTM [14], and fNIRSNet [13] maintain the original network structure with minimal alterations, primarily adjusting input and output dimensions for data accommodation and classification purposes. For fNIRS-T [11], following the specifications in [11], [13], Conv S and Conv C kernel sizes are set to 5 × 10 and 1 × 10, respectively.…”

Section: B Training Settingmentioning

confidence: 99%

See 1 more Smart Citation

Comparison of Deep Learning Models on Detection and Classification of Diabetic Retinopathy

Cao

Chen

2021

2021 7th International Conference on Systems and Informatics (ICSAI)

View full text Add to dashboard Cite

Functional near-infrared spectroscopy (fNIRS) is a valuable non-invasive tool for monitoring brain activity. The classification of fNIRS data in relation to conscious activity holds significance for advancing our understanding of the brain and facilitating the development of brain-computer interfaces (BCI). Many researchers have turned to deep learning to tackle the classification challenges inherent in fNIRS data due to its strong generalization and robustness. In the application of fNIRS, reliability is really important, and one mathematical formulation of the reliability of confidence is calibration. However, many researchers overlook the important issue of calibration. To address this gap, we propose integrating calibration into fNIRS field and assess the reliability of existing models. Surprisingly, our results indicate poor calibration performance in many proposed models. To advance calibration development in the fNIRS field, we summarize three practical tips. Through this letter, we hope to emphasize the critical role of calibration in fNIRS research and argue for enhancing the reliability of deep learning-based predictions in fNIRS classification tasks. All data from our experimental process are openly available on GitHub § .

show abstract

fNIRS Based Multi-Class Mental Workload Classification Using Recurrence Plots and CNN-LSTM

Cited by 8 publications

References 16 publications

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

Empirical comparison of deep learning models for fNIRS pain decoding

Comparison of Deep Learning Models on Detection and Classification of Diabetic Retinopathy

Contact Info

Product

Resources

About