Can Automatic Facial Expression Analysis Be Used for Treatment Outcome Estimation in Schizophrenia?

Bishay, Mina; Priebe, Stefan; Patras, Ioannis

doi:10.1109/icassp.2019.8682652

“…A number of previous works use a bag of words approach for gestures and facial expressions [22,23] or use statistical representations of pre-computed features [5,16,43]. Some of the symptoms have a quantitative measure (e.g., reduced gestures), therefore, intuitively, methodologies that implicitly measure quantity of features have achieved state-of-the-art results in these previous works.…”

Section: Affect and Mental Healthmentioning

confidence: 99%

“…We can see that the proposed methodology outperforms previous works across all the evaluated symptoms and scales by a large margin, particularly for PCC, achieving stateof-the-art results. Since the NESS dataset has been annotated by different healthcare professionals, we can compare the PCC achieved by the proposed method against the PCC of the annotators (mental health experts), which has a mean value of 0.85 [5,37] on NESS. We observe that the proposed method achieves a PCC close to that of human experts for the "Total Negative" and "EXP-Total" scores, in this dataset.…”

Section: Comparison To State-of-the-artmentioning

confidence: 99%

“…The length of the video poses a major difficulty for Machine Learning methods, due to GPU memory constraints. To address this issue, two main approaches are employed in the literature, namely, a) estimating sub-segments of the long videos [8,31] and b) precomputing features [5,35,50,51]. For example, in MIMAMO [12] and the work of Peng et al [36] a small number of frames is sampled from each clip.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning from Label Relationships in Human Affect

Foteinopoulou,

Patras

2022

Preprint

Self Cite

0

View full text Add to dashboard Cite

Human affect and mental state estimation in an automated manner, face a number of difficulties, including learning from labels with poor or no temporal resolution, learning from few datasets with little data (often due to confidentiality constraints) and, (very) long, in-the-wild videos. For these reasons, deep learning methodologies tend to overfit, that is, arrive at latent representations with poor generalisation performance on the final regression task. To overcome this, in this work, we introduce two complementary contributions. First, we introduce a novel relational loss for multilabel regression and ordinal problems that regularises learning and leads to better generalisation. The proposed loss uses label vector inter-relational information to learn better latent representations by aligning batch label distances to the distances in the latent feature space. Second, we utilise a two-stage attention architecture that estimates a target for each clip by using features from the neighbouring clips as temporal context. We evaluate the proposed methodology on both continuous affect and schizophrenia severity estimation problems, as there are methodological and contextual parallels between the two. Experimental results demonstrate that the proposed methodology outperforms the baselines that are trained using the supervised regression loss, as well as pre-training the network architecture with an unsupervised contrastive loss. In the domain of schizophrenia, the proposed methodology outperforms previous state-of-the-art by a large margin, achieving a PCC of up to 78% performance close to that of human experts (85%) and much higher than previous works (uplift of up to 40%). In the case of affect recognition, we outperform previous vision-based methods in terms of CCC on both the OMG and the AMIGOS datasets. Specifically for AMIGOS, we outperform previous SoTA CCC for both arousal and valence by 9% and 13% respectively, and in the OMG dataset we outperform previous vision works by up to 5% for both arousal and valence.

show abstract

“…The length of the video poses a major difficulty for Machine Learning methods, due to GPU memory constraints. To address this issue, two main approaches are employed in the literature, namely, a) estimating sub-segments of the long videos [8,31] and b) precomputing features [5,35,50,51]. For example, in MIMAMO [12] and the work of Peng et al [36] a small number of frames is sampled from each clip.…”

Section: Introductionmentioning

confidence: 99%

Learning from Label Relationships in Human Affect

Foteinopoulou

¹

,

Patras

²

2022

Proceedings of the 30th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Human affect and mental state estimation in an automated manner, face a number of difficulties, including learning from labels with poor or no temporal resolution, learning from few datasets with little data (often due to confidentiality constraints) and, (very) long, in-the-wild videos. For these reasons, deep learning methodologies tend to overfit, that is, arrive at latent representations with poor generalisation performance on the final regression task. To overcome this, in this work, we introduce two complementary contributions. First, we introduce a novel relational loss for multilabel regression and ordinal problems that regularises learning and leads to better generalisation. The proposed loss uses label vector inter-relational information to learn better latent representations by aligning batch label distances to the distances in the latent feature space. Second, we utilise a two-stage attention architecture that estimates a target for each clip by using features from the neighbouring clips as temporal context. We evaluate the proposed methodology on both continuous affect and schizophrenia severity estimation problems, as there are methodological and contextual parallels between the two. Experimental results demonstrate that the proposed methodology outperforms the baselines that are trained using the supervised regression loss, as well as pre-training the network architecture with an unsupervised contrastive loss. In the domain of schizophrenia, the proposed methodology outperforms previous state-of-the-art by a large margin, achieving a PCC of up to 78% performance close to that of human experts (85%) and much higher than previous works (uplift of up to 40%). In the case of affect recognition, we outperform previous vision-based methods in terms of CCC on both the OMG and the AMIGOS datasets. Specifically for AMIGOS, we outperform previous SoTA CCC for both arousal and valence by 9% and 13% respectively, and in the OMG dataset we outperform previous vision works by up to 5% for both arousal and valence.

show abstract

“…Understanding facial expressions are quite important to analyze humans' emotions and non-verbal communications. Automatic Facial Expression Analysis (AFEA) has been an active research area in Computer Vision, as it has gained popularity in several applications like ad testing [1,2,3], driver state monitoring [4,5], and health care [6,7,8]). In ad testing, analyzing customers' facial responses gives traders insights about customers engagement, liking, and purchase intent [1].…”

Section: Introductionmentioning

confidence: 99%

Automatic Detection of Sentimentality from Facial Expressions

Bishay

¹

,

Turcot

²

,

Page

³

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

1

0

View full text Add to dashboard Cite

Emotion recognition has received considerable attention from the Computer Vision community in the last 20 years. However, most of the research focused on analyzing the six basic emotions (e.g. joy, anger, surprise), with a limited work directed to other affective states. In this paper, we tackle sentimentality (strong feeling of heartwarming or nostalgia), a new emotional state that has few works in the literature, and no guideline defining its facial markers. To this end, we first collect a dataset of 4.9K videos of participants watching some sentimental and non-sentimental ads, and then we label the moments evoking sentimentality in the ads. Second, we use the ad-level labels and the facial Action Units (AUs) activation across different frames for defining some weak framelevel sentimentality labels. Third, we train a Multilayer Perceptron (MLP) using the AUs activation for sentimentality detection. Finally, we define two new ad-level metrics for evaluating our model performance. Quantitative and qualitative results show promising results for sentimentality detection. To the best of our knowledge this is the first work to address the problem of sentimentality detection.

show abstract

Can Automatic Facial Expression Analysis Be Used for Treatment Outcome Estimation in Schizophrenia?

Cited by 5 publications

References 21 publications

Learning from Label Relationships in Human Affect

Learning from Label Relationships in Human Affect

Learning from Label Relationships in Human Affect

Automatic Detection of Sentimentality from Facial Expressions

Contact Info

Product

Resources

About