Multimodal Depression Detection: Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors

Alghowinem, Sharifa; Goecke, Roland; Wagner, Michael; Epps, Julien; Hyett, Matthew P.; Parker, Gordon; Breakspear, Michael

doi:10.1109/taffc.2016.2634527

Cited by 125 publications

(81 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They utilize feature and decision level fusion techniques to perform affective computing. In [2], authors utilize paralinguistic, head pose and eye gaze fixations for multimodal depression detection. With the help of statistical tests on the selected features the inference engine will classify the subjects into depressed and healthy categories.…”

Section: Multimodal Approaches For Distress Detectionmentioning

confidence: 99%

Multi-level Attention Network using Text, Audio and Video for Depression Prediction

Ray

Kumar

Reddy

et al. 2019

Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop

View full text Add to dashboard Cite

Depression has been the leading cause of mental-health illness worldwide. Major depressive disorder (MDD), is a common mental health disorder that affects both psychologically as well as physically which could lead to loss of lives. Due to the lack of diagnostic tests and subjectivity involved in detecting depression, there is a growing interest in using behavioural cues to automate depression diagnosis and stage prediction. The absence of labelled behavioural datasets for such problems and the huge amount of variations possible in behaviour makes the problem more challenging. This paper presents a novel multi-level attention based network for multimodal depression prediction that fuses features from audio, video and text modalities while learning the intra and inter modality relevance. The multi-level attention reinforces overall learning by selecting the most influential features within each modality for the decision making. We perform exhaustive experimentation to create different regression models for audio, video and text modalities. Several fusions models with different configurations are constructed to understand the impact of each feature and modality. We outperform the current baseline by 17.52% in terms of root mean squared error.

show abstract

Section: Multimodal Approaches For Distress Detectionmentioning

confidence: 99%

Multi-level Attention Network using Text, Audio and Video for Depression Prediction

Ray

Kumar

Reddy

et al. 2019

Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop

View full text Add to dashboard Cite

show abstract

“…Results achieved in [28] showed that eye gaze, when combined with speech as part of a feature fusion, single support vector regression system, could improve arousal prediction compared to that of unimodal speech (3.5% relative performance improvement), while model fusion improved valence prediction compared to unimodal speech (19.5% rela-tive performance improvement). Psychopathological affective computing work incorporating eye-based features as part of multimodal approaches include post traumatic stress disorder estimation [29] and depression recognition [30], [31].…”

Section: Related Workmentioning

confidence: 99%

Eye-based Continuous Affect Prediction

O'Dwyer

Murray

Flynn

2019

2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

Eye-based information channels include the pupils, gaze, saccades, fixational movements, and numerous forms of eye opening and closure. Pupil size variation indicates cognitive load and emotion, while a person's gaze direction is said to be congruent with the motivation to approach or avoid stimuli. The eyelids are involved in facial expressions that can encode basic emotions. Additionally, eye-based cues can have implications for human annotators of emotions or feelings. Despite these facts, the use of eye-based cues in affective computing is in its infancy, however, and this work is intended to start to address this. Eye-based feature sets, incorporating data from all of the aforementioned information channels, that can be estimated from video are proposed. Feature set refinement is provided by way of continuous arousal and valence learning and prediction experiments on the RECOLA validation set. The eye-based features are then combined with a speech feature set to provide confirmation of their usefulness and assess affect prediction performance compared with group-of-humans-level performance on the RECOLA test set. The core contribution of this paper, a refined eye-based feature set, is shown to provide benefits for affect prediction. It is hoped that this work stimulates further research into eye-based affective computing.

show abstract

“…Albeit the notable advantages, in the evaluation phase, most of the multimodal emotion recognition systems require the synchronous presence of all modalities that are • J. Han employed in the previous training phase [12]- [16]. This severely impedes their application in real life, since it is a common case that information from some particular modalities is missing.…”

Section: Introductionmentioning

confidence: 99%

EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings

Ren

Schuller

2021

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Despite remarkable advances in emotion recognition, they are severely restrained from either the essentially limited property of the employed single modality, or the synchronous presence of all involved multiple modalities. Motivated by this, we propose a novel crossmodal emotion embedding framework called EmoBed, which aims to leverage the knowledge from other auxiliary modalities to improve the performance of an emotion recognition system at hand. The framework generally includes two main learning components, i. e., joint multimodal training and crossmodal training. Both of them tend to explore the underlying semantic emotion information but with a shared recognition network or with a shared emotion embedding space, respectively. In doing this, the enhanced system trained with this approach can efficiently make use of the complementary information from other modalities. Nevertheless, the presence of these auxiliary modalities is not demanded during inference. To empirically investigate the effectiveness and robustness of the proposed framework, we perform extensive experiments on the two benchmark databases RECOLA and OMG-Emotion for the tasks of dimensional emotion regression and categorical emotion classification, respectively. The obtained results show that the proposed framework significantly outperforms related baselines in monomodal inference, and are also competitive or superior to the recently reported systems, which emphasises the importance of the proposed crossmodal learning for emotion recognition.

show abstract

Multimodal Depression Detection: Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors

Cited by 125 publications

References 41 publications

Multi-level Attention Network using Text, Audio and Video for Depression Prediction

Multi-level Attention Network using Text, Audio and Video for Depression Prediction

Eye-based Continuous Affect Prediction

EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings

Contact Info

Product

Resources

About