Multisubject “Learning” for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures

Abstract

An accurate measure of mental workload level has diverse neuroergonomic applications ranging from brain computer interfacing to improving the efficiency of human operators. In this study, we integrated electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), and physiological measures for the classification of three workload levels in an n-back working memory task. A significantly better than chance level classification was achieved by EEG-alone, fNIRS-alone, physiological alone, and EEG+fNIRS based approaches. The results confirmed our previous finding that integrating EEG and fNIRS significantly improved workload classification compared to using EEG-alone or fNIRS-alone. The inclusion of physiological measures, however, does not significantly improves EEG-based or fNIRS-based workload classification. A major limitation of currently available mental workload assessment approaches is the requirement to record lengthy calibration data from the target subject to train workload classifiers. We show that by learning from the data of other subjects, workload classification accuracy can be improved especially when the amount of data from the target subject is small.

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

“…Brain Sci. 2020, 10, x FOR PEER REVIEW 3 of 20 respectively), a hybrid brain data incorporates more information and enabling higher mental decoding accuracy [43] confirming earlier findings [47]. Specifically, in [43] we showed that body physiological measures (heart rate and breathing) did not contribute any new information to fNIRS + EEG based classification of cognitive workload.…”

“…The measurement of neural correlates of cognitive and affective processes using concurrent EEG and fNIRS, multimodal functional neuroimaging, has seen growing interest [43][44][45][46]. As fNIRS and EEG measure complementary aspects of brain activity (hemodynamic and electrophysiological, respectively), a hybrid brain data incorporates more information and enabling higher mental decoding accuracy [43] confirming earlier findings [47]. Specifically, in [43] we showed that body physiological measures (heart rate and breathing) did not contribute any new information to fNIRS + EEG based classification of cognitive workload.…”

Multimodal Physiological Signals for Workload Prediction in Robot-assisted Surgery

“…An F1 score of 0.811 was achieved when incorporating all sensor modalities. This value falls within the upper range of classification accuracies reported in domains outside of RAS, which varied from 45% to 90% [55,[87][88][89][90]. These differences may be due to experiment designs, baseline selections, and task demands.…”

“…This suggests that EEG is the most predictive modality for characterizing workload levels. Other studies in domains outside of RAS [55,[87][88][89][90] also concluded that EEG was the salient modality for workload characterization. In RAS, EEG may be especially reliable due to the design of the dVSS.…”

“…Baseline indicates how ground-truth values for the workload was determined. Although the raw groundtruth workload values have a continuous range, many studies discretized the raw scores into two [31], three [53,55], or four levels [34,56] for training the classification algorithms to predict workload. Subjects describes the sample size and background of the participants.…”