Classification of emotional states via transdermal cardiovascular spatiotemporal facial patterns using multispectral face videos

Shvimmer, Shaul; Simhon, Rotem; Michael, Gilad; Yitzhaky, Yitzhak

doi:10.1038/s41598-022-14808-4

Cited by 4 publications

(25 citation statements)

References 39 publications

(53 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following that, in recent years, research has attempted to develop novel methods for the assessment of a person’s emotional state based on rPPG, mainly from visual cameras (VIS) [ 17 , 18 , 19 , 20 , 21 ]; this approach eliminates the need for uncomfortable sensors and offers the unique advantage of enabling spatial physiological measurement and visualization of the peripheral signals using only one sensor. However, the majority of studies in which PPG signals are extracted from facial video recordings involve averaging all signals from the entire face or specific predefined regions [ 18 , 19 , 20 , 21 ].…”

Section: Introductionmentioning

confidence: 99%

“…Recognizing this limitation, which can be found in a recent work [ 17 ], we proposed a machine learning model to predict emotional states based on physiological changes across the face, which can be related to spatial rPPG methods. This research extracted transdermal spatiotemporal features based on the maximum and minimum values of the spatial rPPG signal obtained from three cameras, which included the visual spectrum range, the near-infrared (NIR) range, and the long-wave infrared (LWIR) thermal range.…”

Section: Introductionmentioning

confidence: 99%

“…In the current study, we aim to improve, simplify, and validate the previous work [ 17 ]. As in [ 17 ], we pursued a “true” contactless (i.e., remotely) emotion classification from short (four seconds) multispectral facial videos of subjects, who passively viewed emotion-eliciting videos, and classified an individual’s emotional state based on features extracted from pulsatile images, which do not rely on stereotypical facial expressions. Validation of the ability to classify emotional states in our dataset was motivated by the fact that emotions in this dataset are hardly expressed.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Talala,

Shvimmer,

Simhon

et al. 2024

Sensors

Self Cite

View full text Add to dashboard Cite

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants’ emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Talala,

Shvimmer,

Simhon

et al. 2024

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Variations in facial expressions result from the conscious and subconscious processing of various internal and external stimuli, including any contextual biases and the interaction between past and present experiences [2]. Patterns of facial muscle movements provide reliable models for automated recognition of human affective states [3]- [5]. Several complex and difficult to identify facial expressions of affective states have also been modelled [5], [6].…”

Section: Introductionmentioning

confidence: 99%

“…Humans understand the dynamic and continuous nature of emotions and affective states using their collective experiences [3]. Given the complexities of emotion elicitation and the psychology behind it, one cannot simply look at emotions as static occurrences in time [5], [7].…”

Section: Introductionmentioning

confidence: 99%

A Hierarchical Separation and Classification Network for Dynamic Micro-Expression Classification

Vice¹,

Khan²,

Tan³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p> Models of seven discrete facial expressions are built on macro-level facial muscle variations for separating distinct affective states. We propose a step-wise Hierarchical Separation and Classification Network (HSCN) that discovers dynamic and continuous macro- and micro-level variations in facial expressions. The HSCN first invokes an unsupervised cosine similarity-based separation method on continuous facial expression data and extracts twenty-one dynamic expression classes from the seven common discrete affective states. Separation between the clusters is then optimised for discovering the macro-level changes in facial muscle activations followed by splitting the upper and lower facial regions for realising and modelling changes pertaining to upper and lower facial muscle activations. A linear discriminant space is developed for clustering the upper and lower facial images on the basis of similar muscular activation patterns. Actual dynamic data and linear discriminant features are mapped for developing a rule-based expert system that would facilitate classification of twenty-one upper and twenty-one lower facial micro-expressions. Using the random forest algorithm, classification accuracies of 76.11\% were observed for dynamic macro-level facial expression classification. A support vector machine provided 73.63\% and 87.68\% accuracies respectively while classifying upper and lower facial micro-expressions. This work provides a novel framework for the dynamic assessment of affective states. Reported methods and results also provide new insight into the dynamic analysis of facial expressions of affective states. </p>

show abstract

A Hierarchical Separation and Classification Network for Dynamic Microexpression Classification

Vice,

Khan,

Tan

et al. 2024

IEEE Trans. Comput. Soc. Syst.

View full text Add to dashboard Cite

Macrolevel facial muscle variations, as used for building models of seven discrete facial expressions, suffice when distinguishing between macrolevel human affective states but won't discretise continuous and dynamic microlevel variations in facial expressions. We present a hierarchical separation and classification network (HSCN) for discovering dynamic, continuous, and macro-and microlevel variations in facial expressions of affective states. In the HSCN, we first invoke an unsupervised cosine similarity-based separation method on continuous facial expression data to extract twenty-one dynamic facial expression classes from the seven common discrete affective states. The between-clusters separation is then optimized for discovering the macrolevel changes resulting from facial muscle activations. A following step in the HSCN separates the upper and lower facial regions for realizing changes pertaining to upper and lower facial muscle activations. Data from the two separated facial regions are then clustered in a linear discriminant space using similarities in muscular activation patterns. Next, the actual dynamic expression data are mapped onto discriminant features for developing a rule-based expert system that facilitates classifying twenty-one upper and twenty-one lower microexpressions. Invoking the random forest algorithm would classify twenty-one macrolevel facial expressions with 76.11% accuracy. A support vector machine (SVM), used separately on upper and lower facial regions in tandem, could classify them with respective accuracies of 73.63% and 87.68%. This work demonstrates a novel and effective method of dynamic assessment of affective states. The HSCN further demonstrates that facial muscle variations gathered from either upper, lower, or full-face would suffice classifying affective states. We also provide new insight into discovery of microlevel facial muscle variations and their utilization in dynamic assessment of facial expressions of affective states.

show abstract

Classification of emotional states via transdermal cardiovascular spatiotemporal facial patterns using multispectral face videos

Cited by 4 publications

References 39 publications

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

A Hierarchical Separation and Classification Network for Dynamic Micro-Expression Classification

A Hierarchical Separation and Classification Network for Dynamic Microexpression Classification

Contact Info

Product

Resources

About