EEG-Based 3D Visual Fatigue Evaluation Using CNN

Abstract

Visual fatigue evaluation plays an important role in applications such as virtual reality since the visual fatigue symptoms always affect the user experience seriously. Existing visual evaluation methods require hand-crafted features for classification, and conduct feature extraction and classification in a separated manner. In this paper, we conduct a designed experiment to collect electroencephalogram (EEG) signals of various visual fatigue levels, and present a multi-scale convolutional neural network (CNN) architecture named MorletInceptionNet to detect visual fatigue using EEG as input, which exploits the spatial-temporal structure of multichannel EEG signals. Our MorletInceptionNet adopts a joint space-time-frequency features extraction scheme in which Morlet wavelet-like kernels are used for time-frequency raw feature extraction and inception architecture are further used to extract multi-scale temporal features. Then, the multi-scale temporal features are concatenated and fed to the fully connected layer for visual fatigue evaluation using classification. In experiment evaluation, we compare our method with five state-of-the-art methods, and the results demonstrate that our model achieve overally the best performance better performance for two widely used evaluation metrics, i.e., classification accuracy and kappa value. Furthermore, we use input-perturbation network-prediction correlation maps to conduct in-depth analysis into the reason why the proposed method outperforms other methods. The results suggest that our model is sensitive to the perturbation of β (14-30 Hz) and γ (30-40 Hz) bands. Furthermore, their spatial patterns are of high correlation with that of the corresponding power spectral densities which are used as evaluation features traditionally. This finding provides evidence of the hypothesis that the proposed model can learn the joint time-frequency-space features to distinguish fatigue levels automatically.2 of 18 ideally indicators. Since mental fatigue is supposed as one aspect of 3D visual fatigue, these features can hardly represent psychological changes [7].Recently, researches have opened up the possibility for visual fatigue assessment with noninvasive approaches based on electroencephalography (EEG) [1]. EEG signals record integration of spontaneous electrical activities of a large number of brain cells from the scalp, which is closely related with mental and physical states [8]. All event-related potentials in EEG are limited in duration and in frequency. The majority of events activate distinct brain regions [9]. This means researchers can extract features from space, time, and frequency dimensions of the EEG data to build efficient classification models for visual fatigue evaluation. Traditionally, the most popular features for visual fatigue evaluation are time-frequency features (e.g., power spectral), which can be obtained by time-frequency transformation (e.g., wavelet transformation). Power spectral of various frequency bands are proven to be related with visual fatigue l...

EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications

“…A CNN has also been used in other EEG-based fatigue recognition and evaluation applications. Yue and Wang [100] applied various fatigue levels' EEG signals to their multiscale CNN architecture, named "MorletInceptionNet," for visual fatigue evaluation. This framework uses a spacetime-frequency combined feature extraction strategy to extract raw features, after which multiscale temporal features are extracted by an inception architecture.…”

Explainable Convolutional Neural Network to Investigate the Age-Related Changes in Multi-Order Functional Connectivity

“…We chose correlation as a measure of FC rather than using other analytic techniques such as coherence, phase locking value, and phase lag index [57]. Such methods can measure EEG connectivity considering nonlinearity and are less sensitive to volume conduction than the correlation [47]. However, they have higher computational complexity and are less straightforward compared with the correlation method.…”

A review of artificial intelligence for EEG‐based brain−computer interfaces and applications

“…It combines with dense layers for spatial‐temporal EEG information process and classification, which achieved 97% accuracy in the classification of EEG driving fatigue levels. Another recent study [22] also assessed diverse fatigue levels in a visual EEG experiment. It proposed a system of multi‐scale CNN that uses a space‐time-frequency combined features extraction strategy to extract raw EEG signals from an inception structure.…”