Measuring Mental Workload with EEG+fNIRS

Abstract

We studied the capability of a Hybrid functional neuroimaging technique to quantify human mental workload (MWL). We have used electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) as imaging modalities with 17 healthy subjects performing the letter n-back task, a standard experimental paradigm related to working memory (WM). The level of MWL was parametrically changed by variation of n from 0 to 3. Nineteen EEG channels were covering the whole-head and 19 fNIRS channels were located on the forehead to cover the most dominant brain region involved in WM. Grand block averaging of recorded signals revealed specific behaviors of oxygenated-hemoglobin level during changes in the level of MWL. A machine learning approach has been utilized for detection of the level of MWL. We extracted different features from EEG, fNIRS, and EEG+fNIRS signals as the biomarkers of MWL and fed them to a linear support vector machine (SVM) as train and test sets. These features were selected based on their sensitivity to the changes in the level of MWL according to the literature. We introduced a new category of features within fNIRS and EEG+fNIRS systems. In addition, the performance level of each feature category was systematically assessed. We also assessed the effect of number of features and window size in classification performance. SVM classifier used in order to discriminate between different combinations of cognitive states from binary- and multi-class states. In addition to the cross-validated performance level of the classifier other metrics such as sensitivity, specificity, and predictive values were calculated for a comprehensive assessment of the classification system. The Hybrid (EEG+fNIRS) system had an accuracy that was significantly higher than that of either EEG or fNIRS. Our results suggest that EEG+fNIRS features combined with a classifier are capable of robustly discriminating among various levels of MWL. Results suggest that EEG+fNIRS should be preferred to only EEG or fNIRS, in developing passive BCIs and other applications which need to monitor users' MWL.

Load-Dependent Relationships between Frontal fNIRS Activity and Performance: A Data-Driven PLS Approach

“…The initial, confirmatory aim of this study was to further validate the use of fNIRS for measuring cognitive load with a large sample and utilizing recently developed robust statistical tools. Though a number of previous fNIRS studies have examined prefrontal activity using attention demanding working memory tasks such as the N-back (Aghajani et al, 2017;Ayaz et al, 2012;Fishburn et al, 2014;Kuruvilla et al, 2013;Sato et al, 2013), recent work has demonstrated that due to the unique statistical properties of fNIRS, the standard analysis approach (based on fMRI) can severely inflate the false positive rate (Huppert, 2016). In addition, discrepancies between studies that may be related to task performance have been demonstrated across a number of studies.…”

Multimodal Physiological Signals for Workload Prediction in Robot-assisted Surgery

“…Intraoperative Objective Non-interruptive Low-cost Questionnaire [10,26]- [29] × × × Expert annotation [14,30] × × Secondary-task [12,13] × Sensor-based [19,28,[31][32][33][34] most widely used questionnaires include NASA Task Load Index (NASA-TLX) [9], SURG-TLX [27], Subjective Workload Assessment Technique (SWAT) [10], and Workload Profile (WP) [29]. Some studies have implemented these surveys intraoperatively, but these surveys interrupt operative flow.…”

“…Baseline indicates how ground-truth values for the workload was determined. Although the raw groundtruth workload values have a continuous range, many studies discretized the raw scores into two [31], three [53,55], or four levels [34,56] for training the classification algorithms to predict workload. Subjects describes the sample size and background of the participants.…”

“…Specifically, subjective perception was commonly used to measure perceived workload as ground truth (e.g., NASA-TLX [32,33], SURG-TLX [28], or general surveys [54]). Other studies used the underlying task difficulty to determine the workload level (e.g., using level of n-back [31,34] and intrinsic task parameters [77]). Both approaches had strengths and weaknesses.…”

On the Use of Machine Learning for EEG-Based Workload Assessment: Algorithms Comparison in a Realistic Task

“…In these cases, the levels of accuracy reached are generally very high, greater than 80% [18], [22], [24]- [29]. Much less are the examples of multiclass-classification [17], whose highest number of workload levels classified has been 7 [30] and, almost all, have been obtained by means of n-back and arithmetic tasks in a laboratory context [31], [32]. In this context, the majority of methods used to define the level of mental workload of a subject are supervised machine learning techniques.…”