Expression recognition with deep features extracted from holistic and part-based models

Abstract

Facial expression recognition aims to accurately interpret facial muscle movements in affective states (emotions). Previous studies have proposed holistic analysis of the face, as well as the extraction of features pertained only to specific facial regions towards expression recognition. While classically the latter have shown better performances, we here explore this in the context of deep learning. In particular, this work provides a performance comparison of holistic and part-based deep learning models for expression recognition. In addition, we showcase the effectiveness of skip connections, which allow a network to infer from both low and high-level feature maps. Our results suggest that holistic models outperform part-based models, in the absence of skip connections. Finally, based on our findings, we propose a data augmentation scheme, which we incorporate in a part-based model. The proposed multi-face multi-part (MFMP) model leverages the wide information from part-based data augmentation, where we train the network using the facial parts extracted from different face samples of the same expression class. Extensive experiments on publicly available datasets show a significant improvement of facial expression classification with the proposed MFMP framework.

A dual‐channel ensembled deep convolutional neural network for facial expression recognition in the wild

“…Table 2 summarizes the comparison report of the proposed DCNNs with the existing CNNs designed to enhance the inference speed 12,14,15,43,58 and recognition accuracy of the FER systems in the complex real-world conditions. [23][24][25]27,32,35,37,40,42,45,50,60,63,66,75 On the validation set of the RAF-DB dataset, the baseline deep locality-preserving CNN (DLP-CNN) introduced by Li et al 66 has achieved recognition accuracy of 84.13%. In the last 5 years, researchers developed different neural networks that have surpassed the baseline classification accuracy on the RAF-DB dataset.…”

“…While both $$$ {\mathrm{DCNN}}_S $$$ and DCE‐DCNN models classified the image into the sadness class. Table 2 summarizes the comparison report of the proposed DCNNs with the existing CNNs designed to enhance the inference speed 12,14,15,43,58 and recognition accuracy of the FER systems in the complex real‐world conditions 23‐25,27,32,35,37,40,42,45,50,60,63,66,75 …”

“…The neural network proposed by Nguyen et al 20 for the FER task has utilized a fusion of facial features extracted using three multilevel CNN (ML‐CNN) built on top of the 18‐layered custom VGG CNN. In another work, Happy et al 45 investigated the use of holistic and part‐based facial features extracted using four VGG‐16 modified CNNs with/without skip connections.…”

FaceTopoNet: Facial Expression Recognition Using Face Topology Learning

“…While many early deep FER approaches such as [12] and [13] focused on holistic representations of the face, many newer methods have shifted focus to part-based approaches to describe the face as distinct [14] or disentangled components [15]. The reason for this focus toward part-based methods is that certain regions of the face such as mouth or periocular areas have been shown to play different roles in reflecting different expressions [16].…”

“…The reason for this focus toward part-based methods is that certain regions of the face such as mouth or periocular areas have been shown to play different roles in reflecting different expressions [16]. Part-based FER techniques typically analyze face regions separately to extract local features, which are then merged to produce global representations [14]. As a result, given that graph-based methods generally lend themselves well to part-based problems, they have recently gained popularity for FER [17]- [22].…”