Deep learning vs. kernel methods: Performance for emotion prediction in videos

Abstract

Recently, mainly due to the advances of deep learning, the performances in scene and object recognition have been progressing intensively. On the other hand, more subjective recognition tasks, such as emotion prediction, stagnate at moderate levels. In such context, is it possible to make affective computational models benefit from the breakthroughs in deep learning? This paper proposes to introduce the strength of deep learning in the context of emotion prediction in videos. The two main contributions are as follow: (i) a new dataset, composed of 30 movies under Creative Commons licenses, continuously annotated along the induced valence and arousal axes (publicly available) is introduced, for which (ii) the performance of the Convolutional Neural Networks (CNN) through supervised finetuning, the Support Vector Machines for Regression (SVR) and the combination of both (Transfer Learning) are computed and discussed. To the best of our knowledge, it is the first approach in the literature using CNNs to predict dimensional affective scores from videos. The experimental results show that the limited size of the dataset prevents the learning or finetuning of CNNbased frameworks but that transfer learning is a promising solution to improve the performance of affective movie content analysis frameworks as long as very large datasets annotated along affective dimensions are not available.

Emotional interactive movie: adjusting the scenario according to the emotional response of the viewer

“…Different approaches are proposed in the literature to computationally infer emotional information from a film. The approach proposed by [22], based on convolutional neural networks, enables to calculate dimensional affective scores for 1-second video portions, on the arousal and valence dimensions (the model is described in details in [23]). Another approach was proposed by Soleymani et al [24], based on a Bayesian classification framework, to classify film scenes into three affective categories: 'calm', 'positive excited', or 'negative excited'.…”

Recognizing induced emotions of movie audiences: Are induced and perceived emotions the same?

“…Here we focus on the continuous subset of the LIRIS-ACCEDE database, which contains 30 full movies, totalling 442 minutes [20]. During data collection, 10 participants watched each movie once and annotated continuous arousal and valence scores (value range [-1,1]) of the emotions they felt during watching (movie induced emotions).…”

“…[28] and [29]). In fact, Baveye et al [20] built a SVR model using only visual features and achieved best reported CC for this task. However, combining multimodal information has improved performance for a number of other emotion recognition tasks (e.g.…”

“…The best reported CC for arousal is 0.337, for valence is 0.296 [20] 2. CC for arousal is 0.054, for valence is 0.017 [31] English movies listed in Table 1 which contain relatively more dialogue.…”

A comprehensive study on mid-level representation and ensemble learning for emotional analysis of video material

“…Logistic regression is then used to predict sentiments using the generated mid-level representations. More recently, Baveye et al [4] compared CNNs applied on video keyframes against the combination of Support Vector Regression (SVR) and low-level features, and reached the conclusion that CNNs constitute a promising solution. Xu et al [40] perform emotion recognition in videos using a BoW approach, where the dictionary is constructed by clustering features obtained from so-called auxiliary images by means of CNNs.…”