Group-Level Emotion Recognition Using Hybrid Deep Models Based on Faces, Scenes, Skeletons and Visual Attentions

Abstract

This paper presents a hybrid deep learning network submitted to the 6th Emotion Recognition in the Wild (EmotiW 2018) Grand Challenge [9], in the category of group-level emotion recognition. Advanced deep learning models trained individually on faces, scenes, skeletons and salient regions using visual attention mechanisms are fused to classify the emotion of a group of people in an image as positive, neutral or negative. Experimental results show that the proposed hybrid network achieves 78.98% and 68.08% classification accuracy on the validation and testing sets, respectively. These results outperform the baseline of 64% and 61%, and achieved the first place in the challenge.

Automatic Emotion Recognition for Groups: A Review

“…Faces, scenes and skeletons are also analysed with CNNs in [82], where on the face-level the CNN output is fed to an LSTM, and where on the scene-level an attention mask is placed over the image. Attention is also applied in [2], where next to faces, scenes, and skeletons, visual attentions are included (salient regions important for emotion detection, found by neural attention mechanisms) by feeding 16 salient regions to a CNN and LSTM. In the work of [?]…”

“…Faces, scenes, and upper bodies [34], [35] Faces, scenes, and bodies/skeletons [80], [81], [82] Faces, scenes, skeletons, [2], [42] and visual attentions/objects Faces and objects [83] Faces, scenes, and places [24] and scene analysis), or fusion of individual emotions in a bottom-up approach.…”

“…The most popular approach towards fusion of different aspects, is to fuse individual predictions in a weighted manner. This is done in [81], [2], [78], [71], and [77]. A combination of individual predictions is also used in [34], [80], and [42], the latter employing majority voting.…”

“…I N recent years, there has been an increased interest in methods for group emotion monitoring. This can be applied with several different goals in mind, such as surveillance, automated image or video annotation, and event detection [1], [2]. Current methods for group monitoring are often resource intensive and depend on human attention, easily overseeing important cues [3], [4].…”

Automatic Group Cohesiveness Detection With Multi-modal Features

“…Holistic (scene-level) information is shown to be the important component in group-level classification in [10,12,24]. While analyzing the cohesiveness of a group of people, it is essential to understand the environments behind the people, e.g., students in a lecture tend to have a low cohesion level, while a group people standing and protesting at a plaza probably have high cohesiveness.…”