Deep Neural Network Augmentation: Generating Faces for Affect Analysis

Abstract

This paper presents a novel approach for synthesizing facial affect; either in terms of the six basic expressions (i.e., anger, disgust, fear, joy, sadness and surprise), or in terms of valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the emotion activation). The proposed approach accepts the following inputs: i) a neutral 2D image of a person; ii) a basic facial expression or a pair of valencearousal (VA) emotional state descriptors to be generated, or a path of affect in the 2D VA Space to be generated as an image sequence. In order to synthesize affect in terms of VA, for this person, 600, 000 frames from the 4DFAB database were annotated. The affect synthesis is implemented by fitting a 3D Morphable Model on the neutral image, then deforming the reconstructed face and adding the inputted affect, and blending the new face with the given affect into the original image. Qualitative experiments illustrate the generation of realistic images, when the neutral image is sampled from thirteen well known lab-controlled or in-the-wild databases, including Aff-Wild, AffectNet, RAF-DB; comparisons with Generative Adversarial Networks (GANs) show the higher quality achieved by the proposed approach. Then, quantitative experiments are conducted, in which the synthesized images are used for data augmentation in training Deep Neural Networks to perform affect recognition over all databases; greatly improved performances are achieved when compared with state-of-theart methods, as well as with GAN-based data augmentation, in all cases.

3D Face Reconstruction in Deep Learning Era: A Survey

“…The histogram of oriented gradients (HOG) and landmark displacement was used for the feature extraction phase. Kollias et al [ 33 ] proposed a novel technique for synthesising facial expressions and the degree of positive/ negative emotion. Based on the valence-arousal (VA) technique, 600 K frames were annotated from the 4DFAB dataset [ 34 ].…”

Pairwise Ranking Network for Affect Recognition

“…Commonly, attempts of training stronger facial expression recognition models are based on aspects irrelevant to annotations per se. Panda et al propose training on largescale data crawled from the internet [22], while Kollias et al artificially generate facial images [11] to be included in the training set. Studying the temporal dynamics of emotions [1] or applying state-of-the-art deep learning architectures such as Transformers [15] is another line of research.…”

The FaceChannel: A Fast and Furious Deep Neural Network for Facial Expression Recognition

“…Lindt et al [25] report experiments using the VGGFace, a variant of the VGG16 network pre-trained for face identification. Kollias et al [40] proposed a novel training mechanics, where it augmented the training set of the AffectNet using a generative adversarial network (GAN), and obtained the best reported accuracy on this corpus, achieving 0.54 CCC for arousal and 0.62 CCC for valence. Our FaceChannel provides an improved performance when compared to most of these results achieving a CCC of 0.46 for arousal and 0.61 for valence.…”

“…Our FaceChannel provides an improved performance when compared to most of these results achieving a CCC of 0.46 for arousal and 0.61 for valence. Different from the work of Kollias et al [40], we train our model using only the available training set portion, and expect these results to improve when training on an augmented training set.…”