Riesz-based Volume Local Binary Pattern and A Novel Group Expression Model for Group Happiness Intensity Analysis

Abstract

Automatic emotion analysis and understanding has received much attention over the years in affective computing. Recently, there are increasing interests in inferring the emotional intensity of a group of people. For group emotional intensity analysis, feature extraction and group expression model are two critical issues. In this paper, we propose a new method to estimate the happiness intensity of a group of people in an image. Firstly, we combine the Riesz transform and the local binary pattern descriptor, named Riesz-based volume local binary pattern, which considers neighbouring changes not only in the spatial domain of a face but also along the different Riesz faces. Secondly, we exploit the continuous conditional random fields for constructing a new group expression model, which considers global and local attributes. Intensive experiments are performed on three challenging facial expression databases to evaluate the novel feature. Furthermore, experiments are conducted on the HAPPEI database to evaluate the new group expression model with the new feature. Our experimental results demonstrate the promising performance for group happiness intensity analysis.

Analyzing Group-Level Emotion with Global Alignment Kernel based Approach

“…Specifically, 'without global weight sort' means that we used face detector [42] to automatically localize multiple faces and output its subsequent face results according to its search order. Following the experiment protocol of Huang et al [18], we used a 4-fold-cross-validation protocol to analyze the influence of 'global weight sort' to SVM-GAK, 2: Performance comparison of SVM-GAK without using global weight sort, SVM-GAK using 'holistic method' and SVM-GAK based on 'minimal spanning tree algorithm', where mean absolute error is used as a performance metric. The last column is the average mean absolute error of all features corresponding with global weight sort method.…”

“…To alleviate previously mentioned problem in group affective analysis, several hybrid model methods were recently proposed by combining bottom-up and top-down components for group affective analysis. They are categorized into two branches: a group expression model [12], [15], [18] and multi-modal framework [10], [11], [19], [20], [21], [22], [23]. The group expression model encodes multiple faces in a group-level image 1 into a graph structure.…”

“…used the group members together with group members' attributes, such as spontaneous expressions, clothes, age, and gender as the bottom-up component. Huang et al [18] proposed another group expression model for group-level happiness intensity estimation to improve the performance. They referred to the global attributes, such as the effect of neighboring group members, as the top-down component, and the local attributes, such as an individual's feature, as the bottom-up component.…”

“…It means that a large number of clusters in k-means could make the feature very sparse, while a small number could lose the discriminative information. In [18], the graph construction suffered from the false prediction of support vector regression. Additionally, group expression model cannot directly measure the distance between images by using statistical models, such as the latent dirichlet allocation.…”

“…According to our empirical analysis on group expression models and multi-modal frameworks, they lacked of adaptation to varied tasks. For example, group expression model with continuous conditional random fields [18] is not suitable to classify emotion category, as it was originally designed to estimate grouplevel happiness intensity. Moreover, they suffered from heavy computations due to many adjustable parameters.…”