Guided Spatial Transformers for Facial Expression Recognition

Abstract

Spatial Transformer Networks are considered a powerful algorithm to learn the main areas of an image, but still, they could be more efficient by receiving images with embedded expert knowledge. This paper aims to improve the performance of conventional Spatial Transformers when applied to Facial Expression Recognition. Based on the Spatial Transformers’ capacity of spatial manipulation within networks, we propose different extensions to these models where effective attentional regions are captured employing facial landmarks or facial visual saliency maps. This specific attentional information is then hardcoded to guide the Spatial Transformers to learn the spatial transformations that best fit the proposed regions for better recognition results. For this study, we use two datasets: AffectNet and FER-2013. For AffectNet, we achieve a 0.35% point absolute improvement relative to the traditional Spatial Transformer, whereas for FER-2013, our solution gets an increase of 1.49% when models are fine-tuned with the Affectnet pre-trained weights.

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

“…For the facial emotion recognizer, we adjusted the weights of the pre-trained STN on the AffectNet dataset to apply transfer-learning strategies. The trained STN for sentiment recognition on AffectNet reached an accuracy of 70.60%, as we can see in [60]. However, applying Feature-Extraction and max.…”

“…Therefore, we trained the STN again with the same database using seven emotions, the same as RAVDESS except for the 'Calm' emotion. This second model reached an accuracy of 65.90% on the AffectNet database using the same parameters and evaluation strategy as in [60].…”

“…For this reason, we employed this type of architecture to solve facial emotion recognition too. Specifically, we utilized the guided STN pre-trained model on AffectNet for valence recognition using saliency masks from Luna-Jiménez et al [60] and adapted it to solve facial emotion recognition.…”

“…The pre-trained facial emotion recognition model was a modified STN that used saliency maps to capture the principal regions of a face, since the performance of these models improved if the STN had access to more processed information about the relevant figures and shapes that appear in an image, as suggested in the previous work of Luna-Jiménez et al [60].…”

Facial Expression Recognition Using Convolutional Neural Network with Attention Module

“…Where ( , ) is the target coordinates of the normal grid of the output feature map, ( , ) is the source coordinates of the input feature map that defines the sample points, and is the affine transformation matrix which might take various transformations. Normalized height and width coordinates so that #1 $ % & , % & $ 1 are within the spatial boundaries of the output and #1 $ % ( , % ( $ 1 are within the spatial boundaries of the input [20]. We can take this mechanism and combine it with ResNet, as seen in Fig.…”