Multilevel Feature Fusion With 3D Convolutional Neural Network for EEG-Based Workload Estimation

Kwak, Youngchul; Kong, Kyeongbo; Song, Woo Jin; Min, Byoung-Kyong; Kim, Seong Eun

doi:10.1109/access.2020.2966834

Cited by 33 publications

(19 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Importantly, the accuracy reported for cross-task classification is rather low, which is in line with the high data variability we observed. More recent publications suggest that larger data sets in combination with more complex deep-learning algorithms, such as recurrent and convolutional networks or even their combinations [ 45 , 46 ] may be promising alternatives that need further investigation. Moreover, deep learning algorithms are a promising candidate in terms of handling cross-task classification [ 47 ].…”

Section: Discussionmentioning

confidence: 99%

Tracking of Mental Workload with a Mobile EEG Sensor

Kutafina

Heiligers

Popović

et al. 2021

Sensors

View full text Add to dashboard Cite

The aim of the present investigation was to assess if a mobile electroencephalography (EEG) setup can be used to track mental workload, which is an important aspect of learning performance and motivation and may thus represent a valuable source of information in the evaluation of cognitive training approaches. Twenty five healthy subjects performed a three-level N-back test using a fully mobile setup including tablet-based presentation of the task and EEG data collection with a self-mounted mobile EEG device at two assessment time points. A two-fold analysis approach was chosen including a standard analysis of variance and an artificial neural network to distinguish the levels of cognitive load. Our findings indicate that the setup is feasible for detecting changes in cognitive load, as reflected by alterations across lobes in different frequency bands. In particular, we observed a decrease of occipital alpha and an increase in frontal, parietal and occipital theta with increasing cognitive load. The most distinct levels of cognitive load could be discriminated by the integrated machine learning models with an accuracy of 86%.

show abstract

Section: Discussionmentioning

confidence: 99%

Tracking of Mental Workload with a Mobile EEG Sensor

Kutafina

Heiligers

Popović

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Therefore, to include spatiotemporal information in the input data for training the network, we converted 1D EEG signals to 3D EEG tensors. To obtain 3D EEG images, we projected the 3D electrode locations on the scalp into a 2D image with size 16 × 16 using azimuthal equidistant projection as in [34] (Fig. 3(a)).…”

Section: Methodsmentioning

confidence: 99%

“…Because 3D convolution can extract both spatial and temporal information, 3D CNNs achieve superior performance in various areas where input data contain 3D data, such as video recognition and EEG decoding [34], [35], [49], [50]. Therefore, we constructed ESNet and FSNet consisting of three 3D convolutional layers with a rectified linear unit (ReLU) activation function to extract spatiotemporal information from 3D EEG and fNIRS tensors, respectively.…”

Section: B 3d Cnn Structure For Unimodal Signalmentioning

confidence: 99%

FGANet: fNIRS-Guided Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces

Kwak

Song

Kim

2022

IEEE Trans. Neural Syst. Rehabil. Eng.

Self Cite

View full text Add to dashboard Cite

Non-invasive brain-computer interfaces (BCIs) have been widely used for neural decoding, linking neural signals to control devices. Hybrid BCI systems using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have received significant attention for overcoming the limitations of EEG-and fNIRS-standalone BCI systems. However, most hybrid EEG-fNIRS BCI studies have focused on late fusion because of discrepancies in their temporal resolutions and recording locations. Despite the enhanced performance of hybrid BCIs, late fusion methods have difficulty in extracting correlated features in both EEG and fNIRS signals. Therefore, in this study, we proposed a deep learning-based early fusion structure, which combines two signals before the fully-connected layer, called the fNIRS-guided attention network (FGANet). First, 1D EEG and fNIRS signals were converted into 3D EEG and fNIRS tensors to spatially align EEG and fNIRS signals at the same time point. The proposed fNIRS-guided attention layer extracted a joint representation of EEG and fNIRS tensors based on neurovascular coupling, in which the spatially important regions were identified from fNIRS signals, and detailed neural patterns were extracted from EEG signals. Finally, the final prediction was obtained by weighting the sum of the prediction scores of the EEG and fNIRS-guided attention features to alleviate performance degradation owing to delayed fNIRS response. In the experimental results, the FGANet significantly outperformed the EEG-standalone network.Furthermore, the FGANet has 4.0% and 2.7% higher accuracy than the state-of-the-art algorithms in mental arithmetic and motor imagery tasks, respectively.

show abstract

“…For example, SVC's have been used to classify different memory workloads, using the n-back task [38], with excellent results. In addition CNN's have been shown to have high classification accuracies for neuroscience applications, where a typical processing chain will convert 1-D EEG signals to 3D EEG images and enable a 3D CNN to learn the spectral and spatial information over the scalp [39]. However, these techniques are considered to be black-box techniques, which generally lack transparency and explainability, limiting psychological interpretations and often failing to progress the knowledge space, which is a fundamental requisite of neuroscience.…”

Section: Literature Reviewmentioning

confidence: 99%

EEG Correlates of Driving Performance

Simpson

Rafferty

2022

IEEE Trans. Human-Mach. Syst.

View full text Add to dashboard Cite

Techniques for monitoring human performance traditionally rely on subjective responses and task-specific scoring, yet research suggests EEG could offer multiple performance metrics with high temporal resolution and accuracy that could be leveraged for human-computer interaction purposes. The objective of the presented work is to investigate which EEG responses correlate with task performance and evaluate whether combinations of these produce effective predictive models, facilitating further understanding of the psychological link to performance. A user study was conducted with 32 participants required to negotiate a driving course with the ambition of learning and improving ability on the course during an EEG recording session. EEG was filtered and post-processed to find Power Spectral Density (PSD) in alpha (α), beta (β), delta (δ), and theta (θ) frequency bands, as well as frontal alpha asymmetry (FAA). The initial laps were considered a baseline and an average performance improvement was calculated over the remaining laps in terms of percentage improvement in duration of track traversal. Results demonstrate Event Related Desynchronisation (ERD) with increased task performance in the alpha (p = .000), delta (p = .000), and theta (p = .000) bands, as well as evidence of a relationship between overall change in FAA and task efficiency. A full electrode analysis identifies δF 4 as the optimal for predicting collisions, with efficiency best predicted by a combination of β Oz and δF 4 .

show abstract

Multilevel Feature Fusion With 3D Convolutional Neural Network for EEG-Based Workload Estimation

Cited by 33 publications

References 45 publications

Tracking of Mental Workload with a Mobile EEG Sensor

Tracking of Mental Workload with a Mobile EEG Sensor

FGANet: fNIRS-Guided Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces

EEG Correlates of Driving Performance

Contact Info

Product

Resources

About