Correlated-spaces regression for learning continuous emotion dimensions

Abstract

Adopting continuous dimensional annotations for affective analysis has been gaining rising attention by researchers over the past years. Due to the idiosyncratic nature of this problem, many subproblems have been identified, spanning from the fusion of multiple continuous annotations to exploiting output-correlations amongst emotion dimensions. In this paper, we firstly empirically answer several important questions which have found partial or no answer at all so far in related literature. In more detail, we study the correlation of each emotion dimension (i) with respect to other emotion dimensions, (ii) to basic emotions (e.g., happiness, anger). As a measure for comparison, we use video and audio features. Interestingly enough, we find that (i) each emotion dimension is more correlated with other emotion dimensions rather than with face and audio features, and similarly (ii) that each basic emotion is more correlated with emotion dimensions than with audio and video features. Motivated by these findings, we present a novel regression algorithm (Correlated-Spaces Regression, CSR), inspired by Canonical Correlation Analysis (CCA) which learns outputcorrelations and performs supervised dimensionality reduction and multimodal fusion by (i) projecting features extracted from all modalities and labels onto a common space where their inter-correlation is maximised and (ii) learning mappings from the projected feature space onto the projected, uncorrelated label space.

Representation of facial expression categories in continuous arousal–valence space: Feature and correlation

“…Since surprise and happiness are generally expressed more exaggeratedly than anger and sadness, such as an open mouth, it seems that the correlation of arousal or valence dimension to an emotion is also correlated with the level of expression exaggeration of this emotion. This also agrees with the result in [6] which indicated that anger has lower correlation than happiness when predicted in continuous values using 2-D facial point features.…”

“…A dimensional space can provide insight into the emotional intensity, as well as the similarity and contrast between different categorized emotions. Although emotion dimensions can be inherently more expressive in comparison to categorized emotions, no explicit mapping between the two descriptions has been established [6]. The correlation between dimensions and categorized emotions is more of an abstract and relatively ambiguous correspondence.…”

“…Nicolaou et al [6] is the only study to the best of our knowledge that investigated the correlation between emotion dimensions and categorized emotion using audio and facial features. Each categorized emotion (i.e.…”

“…Further, nearly all the current knowledge of the correlation between continuous emotion dimensions and categorized emotions is directly adopted from psychological, cognitive, or neuroscience studies [4,5]. From these studies, it can be reasonably assumed that negative valence with negative arousal corresponds to sadness or boredom, but this is more of an abstract and relatively ambiguous correspondence and no explicit mapping between the two descriptions has been established [6]. No other work has been found that computes and estimates the correlations of arousal and valence dimensions to categorized emotions using publicly available databases.…”

Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge

“…using Support Vector Regression (SVR) etc. on image features [28], resulted in very poor performance. Hence, we implemented a deep neural network approach as the baseline.…”