Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

Abstract

Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition.

The Future of Psychotherapy Research and Neuroscience: Introducing the 4E/MoBI Approach to the Study of Patient–Therapist Interaction

“…Studying tiny details in controlled lab settings (“micro”) can answer questions like how therapists read facial expressions (holistically or feature-by-feature?) or if specific brain signals arise during therapy (for example, see Haist & Anzures, 2017; Saffaryazdi et al, 2022). However, laboratory tasks may not capture the real-world complexity of therapy.…”

The future of psychotherapy research and neuroscience: Introducing the 4E/MoBI approach to the study of patient- therapist interaction

“…and can specific biomarkers be obtained from such processes? (Saffaryazdi et al, 2022). In contrast, real-world-like experimental scenarios can help us understand some macro-cognitive dynamics (e.g., is brain-to-brain synchrony increased when patient and therapist create novel dyadic-level knowledge?…”

Affective e-learning approaches, technology and implementation model: a systematic review

“…Technological advancements have significantly altered the practice of teaching and learning. Educators all around the world have faced unprecedented pressure to incorporate technology into their instructional methods, particularly during the innovative epidemic coronavirus disease 2019 (COVID- 19). The pandemic interference resulted in the absolute lockdown of all educational levels' learning centers.…”