Causal affect prediction model using a facial image sequence

Abstract

Among human affective behavior research, facial expression recognition research is improving in performance along with the development of deep learning. However, for improved performance, not only past images but also future images should be used along with corresponding facial images, but there are obstacles to the application of this technique to real-time environments. In this paper, we propose the causal affect prediction network (CAPNet), which uses only past facial images to predict corresponding affective valence and arousal. We train CAPNet to learn causal inference between past images and corresponding affective valence and arousal through supervised learning by pairing the sequence of past images with the current label using the Aff-Wild2 dataset. We show through experiments that the well-trained CAPNet outperforms the baseline of the second challenge of the Affective Behavior Analysis in-the-wild (ABAW2) Competition by predicting affective valence and arousal only with past facial images one-third of a second earlier. Therefore, in real-time application, CAPNet can reliably predict affective valence and arousal only with past data.The code is publicly available. 1

SS-MFAR : Semi-supervised Multi-task Facial Affect Recognition

“…This adaptive threshold is used to mark the predictions on input images as confident and non-confident ones. CE is cross entropy loss 5 and is used on the confident predictions of weak and strong augmentations, KL is symmetric kl-divergence 8 between the probability distributions of non-confident weak and strong predictions and the remaining losses are as in MFAR.…”

“…Automatic affect recognition is currently an active area of research and has applications in many areas such as education, gaming, software development, auto motives [1,2], medical care, etc. Many works have dealt with Valence-arousal estimation [3,4,5,6,7], action unit detection [8,9,10], expression classification [] tasks individually. [11] introduced a framework which uses only static images and Multi-Task-Learning (MTL) to learn categorical representations and use them to estimate dimensional representation, but is limited to AffectNet data set [12].…”

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges

“…The third ABAW Competition, to be held in conjunction with the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022 is a continuation of the first [24] and second [32] ABAW Competitions held in conjunction with the IEEE Conference on Face and Gesture Recognition (IEEE FG) 2021 and with the International Conference on Computer Vision (ICCV) 2022, respectively, which targeted dimensional (in terms of valence and arousal) [2][3][4]8,9,11,21,35,39,47,48,50,[54][55][56], categorical (in terms of the basic expressions) [12,15,16,33,36,37,51] and facial action unit analysis and recognition [7,19,20,25,26,40,44,47]. The third ABAW Competition contains four Challenges, which are based on the same in-the-wild database, (i) the uni-task Valence-Arousal Estimation Challenge; (ii) the uni-task Expression Classification Challenge (for the 6 basic expressions plus the neutral state plus the 'other' category that denotes expressions/affective states other than the 6 basic ones); (iii) the uni-task Action Unit Detection Challenge (for 12 action units); (iv) the Multi-Task Learning Challenge (for joint learning and predicting of valence-arousal, 8 expressions -6 basic plus neutral plus 'other'-and 12 action units).…”

Incorporating causality in energy consumption forecasting using deep neural networks

“…Importantly, E-RNN can be combined with various state-of-the-art RNN techniques such as multilayer (Parlos et al, 1994), dropout (Srivastava et al, 2014), LSTM (Hochreiter & Schmidhuber, 1997), and GRU (Chung et al, 2015) and leads to performance gains, without the need for any additional assumptions. Entangled Recurrent Neural Networks have been used in multiple domains such as face image sequencing (Oh et al, 2021), load forecasting (Sriram et al, 2018), and other power system applications (Sriram, 2020). However, as aforementioned, to the best of the authors' knowledge, E-RNNs have not been used in the domain of OR, especially in energy demand forecasting.…”