Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis

Zhang, Zheng; Girard, Jeffrey M.; Wu, Yue; Zhang, Xing; Liu, Peng; Ciftci, Umur Aybars; Canavan, Shaun; Reale, Michael; Horowitz, Andrew; Yang, Huamin; Cohn, Jeffrey F.; Ji, Qiang; Liu, Yin

doi:10.1109/cvpr.2016.374

Cited by 317 publications

(186 citation statements)

References 27 publications

Supporting

Mentioning

172

Contrasting

Order By: Relevance

“…We trained a convolutional neural network (CNN) on data from 160 participants from the BP4D and BP4D+ databases [30, 31]. Subsets of these data have been used in FERA 2017 [29] and 3DFAW [21].…”

Section: Methodsmentioning

confidence: 99%

Automated Affect Detection in Deep Brain Stimulation for Obsessive-Compulsive Disorder

Cohn

Jeni

Ertuğrul

et al. 2018

Proceedings of the 20th ACM International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

Automated measurement of affective behavior in psychopathology has been limited primarily to screening and diagnosis. While useful, clinicians more often are concerned with whether patients are improving in response to treatment. Are symptoms abating, is affect becoming more positive, are unanticipated side effects emerging? When treatment includes neural implants, need for objective, repeatable biometrics tied to neurophysiology becomes especially pressing. We used automated face analysis to assess treatment response to deep brain stimulation (DBS) in two patients with intractable obsessive-compulsive disorder (OCD). One was assessed intraoperatively following implantation and activation of the DBS device. The other was assessed three months post-implantation. Both were assessed during DBS on and o conditions. Positive and negative valence were quantified using a CNN trained on normative data of 160 non-OCD participants. Thus, a secondary goal was domain transfer of the classifiers. In both contexts, DBS-on resulted in marked positive affect. In response to DBS-off, affect flattened in both contexts and alternated with increased negative affect in the outpatient setting. Mean AUC for domain transfer was 0.87. These findings suggest that parametric variation of DBS is strongly related to affective behavior and may introduce vulnerability for negative affect in the event that DBS is discontinued.

show abstract

Section: Methodsmentioning

confidence: 99%

Automated Affect Detection in Deep Brain Stimulation for Obsessive-Compulsive Disorder

Cohn

Jeni

Ertuğrul

et al. 2018

Proceedings of the 20th ACM International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

show abstract

“…[13,47,65]) or dynamic facial expressions (e.g. [2,15,18,44,64,68,69]). Most of these datasets focus on emotional expressions and only a few datasets capture facial dynamics caused by speech.…”

Section: Related Workmentioning

confidence: 99%

Capture, Learning, and Synthesis of 3D Speaking Styles

Cudeiro

Bolkart

Laidlaw

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

225

205

View full text Add to dashboard Cite

Input: speech signal and 3D templateOutput: 3D character animation Figure 1: Given an arbitrary speech signal and a static 3D face mesh as input (left), our model, VOCA outputs a realistic 3D character animation (right). Top: Winston Churchill. Bottom: Actor from Karras et al. [33]. See supplementary video. AbstractAudio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input-even speech in languages other than English-and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is read-ily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at

show abstract

“…Our system analyzed videos from the MMSE-HR database of spontaneous expressions [16]. In one half of the database, subjects are performing in a task ("T10") in which a series of three darts are thrown nearer and nearer past/above their head by the experimental facilitator (to provoke a "fear response") [16]. The moments in time when the darts are thrown are not included as part of the dataset but were inferred from visual inspection of subject responses.…”

Section: Designmentioning

confidence: 99%

“…Research efforts on data such as this [10,11,16] often try to "detect" what emotions subjects were feeling during these videos (are these "Fear" responses? "Nervous" responses?…”

Section: Designmentioning

confidence: 99%

Constructionist steps towards an autonomously empathetic system

Buteau

Lyons

2018

Proceedings of the 20th International Conference on Multimodal Interaction: Adjunct

View full text Add to dashboard Cite

Prior efforts to create an autonomous computer system capable of predicting what a human being is thinking or feeling from facial expression data have been largely based on outdated, inaccurate models of how emotions work that rely on many scientifically questionable assumptions. In our research, we are creating an empathetic system that incorporates the latest provable scientific understanding of emotions: that they are constructs of the human mind, rather than universal expressions of distinct internal states. Thus, our system uses a user-dependent method of analysis and relies heavily on contextual information to make predictions about what subjects are experiencing. Our system's accuracy and therefore usefulness are built on provable ground truths that prohibit the drawing of inaccurate conclusions that other systems could too easily make.

show abstract

Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis

Cited by 317 publications

References 27 publications

Automated Affect Detection in Deep Brain Stimulation for Obsessive-Compulsive Disorder

Automated Affect Detection in Deep Brain Stimulation for Obsessive-Compulsive Disorder

Capture, Learning, and Synthesis of 3D Speaking Styles

Constructionist steps towards an autonomously empathetic system

Contact Info

Product

Resources

About