An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition

Wu, Minchao; Teng, Wei; Fan, Cunhang; Pei, Shengbing; Li, Ping; Lv, Zhao

doi:10.1109/tnsre.2023.3253866

Cited by 27 publications

(10 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multiple human senses can be stimulated to develop emotions through the use of audio–visual information employed in multisensory media studies. The examination of facial expressions or neuro-physiological signals has been the primary focus of databases for the research of affect recognition based on visual modalities [ 22 , 23 , 24 , 25 , 26 ]. Yet, despite the fact that eye movements have been shown to be valuable indicators of affective response [ 18 ], few researchers have concentrated on the creation of relevant databases.…”

Section: Eye-tracking Databases For Emotion Recognitionmentioning

confidence: 99%

eSEE-d: Emotional State Estimation Based on Eye-Tracking Dataset

et al. 2023

View full text Add to dashboard Cite

Affective state estimation is a research field that has gained increased attention from the research community in the last decade. Two of the main catalysts for this are the advancement in the data analysis using artificial intelligence and the availability of high-quality video. Unfortunately, benchmarks and public datasets are limited, thus making the development of new methodologies and the implementation of comparative studies essential. The current work presents the eSEE-d database, which is a resource to be used for emotional State Estimation based on Eye-tracking data. Eye movements of 48 participants were recorded as they watched 10 emotion-evoking videos, each of them followed by a neutral video. Participants rated four emotions (tenderness, anger, disgust, sadness) on a scale from 0 to 10, which was later translated in terms of emotional arousal and valence levels. Furthermore, each participant filled three self-assessment questionnaires. An extensive analysis of the participants’ answers to the questionnaires’ self-assessment scores as well as their ratings during the experiments is presented. Moreover, eye and gaze features were extracted from the low-level eye-recorded metrics, and their correlations with the participants’ ratings are investigated. Finally, we take on the challenge to classify arousal and valence levels based solely on eye and gaze features, leading to promising results. In particular, the Deep Multilayer Perceptron (DMLP) network we developed achieved an accuracy of 92% in distinguishing positive valence from non-positive and 81% in distinguishing low arousal from medium arousal. The dataset is made publicly available.

show abstract

Section: Eye-tracking Databases For Emotion Recognitionmentioning

confidence: 99%

eSEE-d: Emotional State Estimation Based on Eye-Tracking Dataset

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Human emotions can be expressed through speech ( Zhao et al, 2021 ), text ( Wang et al, 2020b ), gestures ( Li Y. -K. et al, 2022 ), and physiological signals ( Wu et al, 2023 ), but facial expressions most intuitively reflect human emotions. The research has found that people would intentionally display certain facial expressions in certain situations, yet when people try to hide their facial expressions in high-stakes situations, it is necessary to interpret facial micro-expressions to determine their true emotional state ( Ekman and Friesen, 1969 ).…”

Section: Introductionmentioning

confidence: 99%

Multi-scale fusion visual attention network for facial micro-expression recognition

et al. 2023

View full text Add to dashboard Cite

IntroductionMicro-expressions are facial muscle movements that hide genuine emotions. In response to the challenge of micro-expression low-intensity, recent studies have attempted to locate localized areas of facial muscle movement. However, this ignores the feature redundancy caused by the inaccurate locating of the regions of interest.MethodsThis paper proposes a novel multi-scale fusion visual attention network (MFVAN), which learns multi-scale local attention weights to mask regions of redundancy features. Specifically, this model extracts the multi-scale features of the apex frame in the micro-expression video clips by convolutional neural networks. The attention mechanism focuses on the weights of local region features in the multi-scale feature maps. Then, we mask operate redundancy regions in multi-scale features and fuse local features with high attention weights for micro-expression recognition. The self-supervision and transfer learning reduce the influence of individual identity attributes and increase the robustness of multi-scale feature maps. Finally, the multi-scale classification loss, mask loss, and removing individual identity attributes loss joint to optimize the model.ResultsThe proposed MFVAN method is evaluated on SMIC, CASME II, SAMM, and 3DB-Combined datasets that achieve state-of-the-art performance. The experimental results show that focusing on local at the multi-scale contributes to micro-expression recognition.DiscussionThis paper proposed MFVAN model is the first to combine image generation with visual attention mechanisms to solve the combination challenge problem of individual identity attribute interference and low-intensity facial muscle movements. Meanwhile, the MFVAN model reveal the impact of individual attributes on the localization of local ROIs. The experimental results show that a multi-scale fusion visual attention network contributes to micro-expression recognition.

show abstract

“…Chakravarthi et al ( 2022 ) proposed an automated CNN-LSTM with the ResNet-152 algorithm to identify emotional states from EEG signals. Additionally, Wu et al ( 2023 ) developed a novel experimental paradigm that allows odors dynamically participate in different stages of video-evoked emotions, to investigate the efficiency of olfactory-enhanced videos in inducing subjects' emotions. Thus, understanding the emotions expressed in an utterance requires a comprehensive understanding of various modalities.…”

Section: Introductionmentioning

confidence: 99%

GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognition

Luo

Wang

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

Emotion recognition plays an essential role in interpersonal communication. However, existing recognition systems use only features of a single modality for emotion recognition, ignoring the interaction of information from the different modalities. Therefore, in our study, we propose a global-aware Cross-modal feature Fusion Network (GCF2-Net) for recognizing emotion. We construct a residual cross-modal fusion attention module (ResCMFA) to fuse information from multiple modalities and design a global-aware module to capture global details. More specifically, we first use transfer learning to extract wav2vec 2.0 features and text features fused by the ResCMFA module. Then, cross-modal fusion features are fed into the global-aware module to capture the most essential emotional information globally. Finally, the experiment results have shown that our proposed method has significant advantages than state-of-the-art methods on the IEMOCAP and MELD datasets, respectively.

show abstract

An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition

Cited by 27 publications

References 51 publications

eSEE-d: Emotional State Estimation Based on Eye-Tracking Dataset

eSEE-d: Emotional State Estimation Based on Eye-Tracking Dataset

Multi-scale fusion visual attention network for facial micro-expression recognition

GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognition

Contact Info

Product

Resources

About