EmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege’s Principle

Mittal, Trisha; Guhan, Pooja; Bhattacharya, Uttaran; Chandra, Rohan; Bera, Aniket; Manocha, Dinesh

doi:10.1109/cvpr42600.2020.01424

Cited by 121 publications

(73 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To counter that effect, we employ both localization and semantic information to evaluate the effect of each object on the primary agent. Mittal et al [15] presented the EmotiCon based on Frege's Context Principle from psychology. Their method included three streams: multiple modalities of faces and gaits, background context and socio-dynamic inter-agent interactions.…”

Section: A Emotion Recognitionmentioning

confidence: 99%

“…Those intermediate features are concatenated and denoted by X m . In the same spirit with [13], [15], in the scene-level stream, we conceal the primary agent by a black box with the purpose of assisting the model to learn only the background information. The features produced from the scene-level stream are denoted by X s .…”

Section: A Algorithm Overviewmentioning

confidence: 99%

“…The optimizer algorithm for the whole model is Stochastic Gradient Descent (SGD) with a batch size of 26. Following [15], [32], our model is trained with the weighted L 2 loss and the modified loss function for multi-modalities which focuses on more accurate stream information while reducing concentration on less trustworthy streams. The total loss can be defined as follows:…”

Section: B Implementation Detailsmentioning

confidence: 99%

“…They tried to build the correlation of all proposed regions by densely relationship while we quantify the interaction of instance objects appearing in the image on the primary agent. Another approach is EmotiCon [15] which learned the context by using Frege's principle. This method included three kinds of contexts: multiple modalities (mixing face and body pose), background context, and socio dynamic inter-agent interactions.…”

Section: Comparison With the State-of-the-artmentioning

confidence: 99%

See 3 more Smart Citations

Context-Aware Emotion Recognition Based on Visual Relationship Detection

et al. 2021

View full text Add to dashboard Cite

Emotion recognition, which is a part of affective computing, draws a lot of attention from researchers because of its broad applications. Unlike previous approaches with the aim to recognize humans' emotional state using facial expression, speech or gesture, some researchers see the potential of the contextual information from the scene. Hence, in addition to the employment of the main subject, the general background data is also considered as the complementary cues for emotion prediction. However, most of the existing works still have some limitations in deeply exploring the scene-level context. In this paper, to fully exploit the essences of context, we propose the emotional state prediction method based on visual relationship detection between the main target and the adjacent objects from the background. Specifically, we utilize both the spatial and semantic features of objects in the scene to calculate the influences of all context-related elements and their properties of impact (positive, negative, or neutral) on the main subject by a modified attention mechanism. After that, the model incorporates those features with scene context and body features of the target person to predict their emotional states. Our experimental results achieve state-of-the-art performance on the CAER-S dataset and competitive results on the EMOTIC benchmark.

show abstract

Section: A Emotion Recognitionmentioning

confidence: 99%

Section: A Algorithm Overviewmentioning

confidence: 99%

Section: B Implementation Detailsmentioning

confidence: 99%

Section: Comparison With the State-of-the-artmentioning

confidence: 99%

See 2 more Smart Citations

Context-Aware Emotion Recognition Based on Visual Relationship Detection

et al. 2021

View full text Add to dashboard Cite

show abstract

“…While human arXiv:2012.09402v1 [cs.CV] 17 Dec 2020 perception typically involves inferring the physical attributes about the humans (detection [5,35,43,50], poses [3,4,8,25,28,41], shape [13,20,29,30], gaze [44] etc. ), interpreting humans involves reasoning about the finer details relating to human activity [6,24,27,48,49], behaviour [26,34], human-object visual relationship detection [23,33,36,37,39,40], and human-object interactions [23,32,33,36,37,39,40,42]. In this work, we investigate the problem of identifying Human-Object Interactions in videos.…”

Section: Introductionmentioning

confidence: 99%

Lighten

Sunkesula

Dabral

Ramakrishnan

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Analyzing the interactions between humans and objects from a video includes identification of the relationships between humans and the objects present in the video. It can be thought of as a specialized version of Visual Relationship Detection, wherein one of the objects must be a human. While traditional methods formulate the problem as inference on a sequence of video segments, we present a hierarchical approach, LIGHTEN, to learn visual features to effectively capture spatio-temporal cues at multiple granularities in a video. Unlike current approaches, LIGHTEN avoids using ground truth data like depth maps or 3D human pose, thus increasing generalization across non-RGBD datasets as well. Furthermore, we achieve the same using only the visual features, instead of the

show abstract

Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping

Bhattacharya

Roncal

Mittal

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

We present an autoencoder-based semi-supervised approach to classify perceived human emotions from walking styles obtained from videos or from motion-captured data and represented as sequences of 3D poses. Given the motion on each joint in the pose at each time step extracted from 3D pose sequences, we hierarchically pool these joint motions in a bottom-up manner in the encoder, following the kinematic chains in the human body. We also constrain the latent embeddings of the encoder to contain the space of psychologically-motivated affective features underlying the gaits. We train the decoder to reconstruct the motions per joint per time step in a top-down manner from the latent embeddings. For the annotated data, we also train a classifier to map the latent embeddings to emotion labels. Our semisupervised approach achieves a mean average precision of 0.84 on the Emotion-Gait benchmark dataset, which contains gaits collected from multiple sources. We outperform current state-of-art algorithms for both emotion recognition and action recognition from 3D gaits by 7% -23% on the absolute.

show abstract

EmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege’s Principle

Cited by 121 publications

References 40 publications

Context-Aware Emotion Recognition Based on Visual Relationship Detection

Context-Aware Emotion Recognition Based on Visual Relationship Detection

Lighten

Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping

Contact Info

Product

Resources

About