Recognition of Affect in the Wild Using Deep Neural Networks

Kollias, Dimitrios; Nicolaou, Mihalis A.; Kotsia, Irene; Zhao, Guoying; Zafeiriou, Stefanos

doi:10.1109/cvprw.2017.247

Cited by 121 publications

(110 citation statements)

References 18 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…CNNs were first applied in [24] to predict dimensional affective scores from videos, but the issue of small samples (raised above) challenged CNN learning. In [25], CNNs were combined with recurrent neural networks to model arousal-valence using the Aff-Wild database [26]. In [27] the authors exploit deep end-to-end trainable networks for recognizing affect in real-world environments.…”

Section: A Video-based Affect Modelingmentioning

confidence: 99%

From Pixels to Affect: A Study on Games and Player Experience

Makantasis

Liapis

Yannakakis

2019

2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

Is it possible to predict the affect of a user just by observing her behavioral interaction through a video? How can we, for instance, predict a user's arousal in games by merely looking at the screen during play? In this paper we address these questions by employing three dissimilar deep convolutional neural network architectures in our attempt to learn the underlying mapping between video streams of gameplay and the player's arousal. We test the algorithms in an annotated dataset of 50 gameplay videos of a survival shooter game and evaluate the deep learned models' capacity to classify high vs low arousal levels. Our key findings with the demanding leave-onevideo-out validation method reveal accuracies of over 78% on average and 98% at best. While this study focuses on games and player experience as a test domain, the findings and methodology are directly relevant to any affective computing area, introducing a general and user-agnostic approach for modeling affect.

show abstract

Section: A Video-based Affect Modelingmentioning

confidence: 99%

From Pixels to Affect: A Study on Games and Player Experience

Makantasis

Liapis

Yannakakis

2019

2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

show abstract

“…Aff-Wild We synthesized 60,135 images from the Aff-Wild database and added those images to the training set of the first Affect-in-the-wild Challenge. The employed network architecture was the AffWildNet (VGG-FACE-GRU) described in [29,30]. Table 2 shows a comparison of the performance of: the VGG-FACE-GRU trained using: i) our approach, ii) StarGAN, and iii) Ganimation; the best performing network, AffWildNet, reported in [29,30]; the winner of the Aff-Wild Challenge [13] (FATAUVA-Net).…”

Section: Experiments On Dimensional Affectmentioning

confidence: 99%

Deep Neural Network Augmentation: Generating Faces for Affect Analysis

et al. 2020

Self Cite

View full text Add to dashboard Cite

This paper presents a novel approach for synthesizing facial affect; either in terms of the six basic expressions (i.e., anger, disgust, fear, joy, sadness and surprise), or in terms of valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the emotion activation). The proposed approach accepts the following inputs: i) a neutral 2D image of a person; ii) a basic facial expression or a pair of valencearousal (VA) emotional state descriptors to be generated, or a path of affect in the 2D VA Space to be generated as an image sequence. In order to synthesize affect in terms of VA, for this person, 600, 000 frames from the 4DFAB database were annotated. The affect synthesis is implemented by fitting a 3D Morphable Model on the neutral image, then deforming the reconstructed face and adding the inputted affect, and blending the new face with the given affect into the original image. Qualitative experiments illustrate the generation of realistic images, when the neutral image is sampled from thirteen well known lab-controlled or in-the-wild databases, including Aff-Wild, AffectNet, RAF-DB; comparisons with Generative Adversarial Networks (GANs) show the higher quality achieved by the proposed approach. Then, quantitative experiments are conducted, in which the synthesized images are used for data augmentation in training Deep Neural Networks to perform affect recognition over all databases; greatly improved performances are achieved when compared with state-of-theart methods, as well as with GAN-based data augmentation, in all cases.

show abstract

“…Recent advances in deep neural networks [11], [13], [14], [15] have been explored in [6], where convolutional (CNN) and convolutional-recurrent (CNN-RNN) neural networks were developed and trained to classify the information in the above Parkinson's database in two categories, i.e., patients and non patients, based on either MRI inputs, or DaT Scan inputs, or together MRI and DaT Scan inputs.…”

Section: Related Workmentioning

confidence: 99%

Predicting Parkinson’s Disease using Latent Information extracted from Deep Neural Networks

Kollia

Stafylopatis

Kollias

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

This paper presents a new method for medical diagnosis of neurodegenerative diseases, such as Parkinson's, by extracting and using latent information from trained Deep convolutional, or convolutional-recurrent Neural Networks (DNNs). In particular, our approach adopts a combination of transfer learning, k-means clustering and k-Nearest Neighbour classification of deep neural network learned representations to provide enriched prediction of the disease based on MRI and/or DaT Scan data. A new loss function is introduced and used in the training of the DNNs, so as to perform adaptation of the generated learned representations between data from different medical environments. Results are presented using a recently published database of Parkinson's related information, which was generated and evaluated in a hospital environment.Index Terms-latent variable information, deep convolutional and recurrent neural networks, transfer learning and domain adaptation, modified loss function, prediction, Parkinson's disease, MRI, DaT Scan data.

show abstract

Recognition of Affect in the Wild Using Deep Neural Networks

Cited by 121 publications

References 18 publications

From Pixels to Affect: A Study on Games and Player Experience

From Pixels to Affect: A Study on Games and Player Experience

Deep Neural Network Augmentation: Generating Faces for Affect Analysis

Predicting Parkinson’s Disease using Latent Information extracted from Deep Neural Networks

Contact Info

Product

Resources

About