Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Sun, Yanjia; Ayaz, Hasan; Akansu, Ali N.

doi:10.3390/brainsci10020085

Cited by 28 publications

(16 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Facial expressions are one of the most commonly used input modalities analyzed to identify emotional state (Sun et al, 2020 ). They are used in many HCI applications (Samadiani et al, 2019 ).…”

Section: Introductionmentioning

confidence: 99%

Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

et al. 2022

View full text Add to dashboard Cite

Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition.

show abstract

Section: Introductionmentioning

confidence: 99%

Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

et al. 2022

View full text Add to dashboard Cite

show abstract

“…A hybrid BCI is expected to achieve better performance and classification accuracy than other conventional systems [26]. EEG and fNIRS measure the complementary characteristics of brain signals, i.e., electrophysiological and hemodynamic aspects, so a hybrid BCI integrates more information producing better results than using individual modalities [27][28][29][30].…”

Section: Introductionmentioning

confidence: 99%

Hemodynamic Response Detection Using Integrated EEG-fNIRS-VPA for BCI

Arif¹,

Khan²,

Javed³

et al. 2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

For BCI systems, it is important to have an accurate and less complex architecture to control a device with enhanced accuracy. In this paper, a novel methodology for more accurate detection of the hemodynamic response has been developed using a multimodal brain-computer interface (BCI). An integrated classifier has been developed for achieving better classification accuracy using two modalities. An integrated EEG-fNIRS-based vector-phase analysis (VPA) has been conducted. An open-source dataset collected at the Technische Universität Berlin, including simultaneous electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals of 26 healthy participants during n-back tests, has been used for this research. Instrumental and physiological noise removal has been done using preprocessing techniques followed by individually detecting activity in both modalities. With resting state threshold circle, VPA has been used to detect a hemodynamic response in fNIRS signals, whereas phase plots for EEG signals have been constructed using Hilbert Transform to detect the activity in each trial. Multiple threshold circles are drawn in the vector plane, where each circle is drawn after task completion in each trial of EEG signal. Finally, both processes are integrated into one vector-phase plot to get combined detection of hemodynamic response for activity. Results of this study illustrate that the combined EEG-fNIRS VPA yields considerably higher average classification accuracy, that is 91.35%, as compared to other classifiers such as support vector machine (SVM), convolutional neural networks (CNN), deep neural networks (DNN) and VPA (with dual-threshold circles) with classification accuracies 82%, 89%, 87% and 86% respectively. Outcomes of this research demonstrate that improved classification performance can be feasibly achieved using multimodal VPA for EEG-fNIRS hybrid data.

show abstract

“…Invasive and noninvasive techniques are used in brain-computer interface (BCI) for the detection and measurement of brain activities using different BCI modalities (Sun et al, 2020;Tortora et al, 2020). Invasive BCI is based upon placing electrodes inside the brain cortex under direct interaction with neurons and hence requires complex surgery, medical conditions, and greater risk of infections (Yoo et al, 2018;Alkawadri, 2019;Romanelli et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain–Computer Interface

Arif¹,

Khan²,

Naseer³

et al. 2021

Front. Hum. Neurosci.

View full text Add to dashboard Cite

A passive brain–computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects’ data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.

show abstract

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Cited by 28 publications

References 73 publications

Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

Hemodynamic Response Detection Using Integrated EEG-fNIRS-VPA for BCI

Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain–Computer Interface

Contact Info

Product

Resources

About