Robust Lightweight Facial Expression Recognition Network with Label Distribution Training

Abstract

This paper presents an efficiently robust facial expression recognition (FER) network, named EfficientFace, which holds much fewer parameters but more robust to the FER in the wild. Firstly, to improve the robustness of the lightweight network, a local-feature extractor and a channel-spatial modulator are designed, in which the depthwise convolution is employed. As a result, the network is aware of local and global-salient facial features. Then, considering the fact that most emotions occur as combinations, mixtures, or compounds of the basic emotions, we introduce a simple but efficient label distribution learning (LDL) method as a novel training strategy. Experiments conducted on realistic occlusion and pose variation datasets demonstrate that the proposed EfficientFace is robust under occlusion and pose variation conditions. Moreover, the proposed method achieves state-of-the-art results on RAF-DB, CAER-S, and AffectNet-7 datasets with accuracies of 88.36%, 85.87%, and 63.70%, respectively, and a comparable result on the AffectNet-8 dataset with an accuracy of 59.89%. The code is public available at https://github.com/zengqunzhao/EfficientFace.

FER-former: Multi-modal Transformer for Facial Expression Recognition

“…RAF-DB dataset is one of the most widely used large-scale realworld FER datasets because it facilitates fair comparisons, in which all images are cropped and do not require any additional preprocessing. Results show that our FaceFormer achieves state-of-the-art performance compared to all other methods, including FER with unconstrained variations (RAN [11], MA-Net [13], IPD-FER [58]), and FER with annotation ambiguity (SCN [30], DMUE [28], KTN [59], EfficientFace [26], SPLDL [29], EASE [32]). In particular, when compared to TransFER [23], the previous best achieved by combining CNN and ViT, FER-former lowers the error rate from 9.09% to 8.7%, a 4.3% improvement.…”

“…Label distribution learning is an intuitive and favoured scheme to reduce the ambiguity. Zhao et al [26] treat the output of an auxiliary ResNet-50 as probability distribution to guide the learning of the other backbone network. Shao et al [29] further adopt an auxiliary network as a label distribution generator to generate label distributions for guiding the backbone network training and selecting easy samples.…”

“…Lots of efforts have been made to address the issue of annotation ambiguity and can be roughly categorized as two mainstream directions: Learning of Soft Label Distribution and Learning from Hard Samples via delicately-designed mechanisms (e.g., attention mechanism or self-supervised strategy). Specifically, i) Instead of hard one-hot supervisory signals, a set of probability distributions with auxiliary priors are assigned to facial images with the goal of driving the deep learning-based models to extract representations with fuzzy boundaries [26]- [29]; ii) Hard samples are selected as outliers and then imposed on extra special processes to suppress their negative impacts on the holistic models' learning [29]- [32]. Even though a series of attempts have been conducted to ease annotation ambiguity, the required auxiliary supervisory signals are still derived from raw one-hot labels and the ambiguity is prone to be inherited.…”

POSTER++: A simpler and stronger facial expression recognition network

“…Savchenko et al [38] first verified the effectiveness of CNNs such as Mo-bileNet [19], EfficientNet [41] and RexNet [15] for FER. Zhao et al proposed an efficient and robust FER network EfficientFace [57] for the analysis of facial expressions in the wild. Nevertheless, convolution-based FER algorithms cannot consider the global information of the image due to the limitation of convolutional local receptive field.…”

ExpresSense: Exploring a Standalone Smartphone to Sense Engagement of Users from Facial Expressions Using Acoustic Sensing

“…Facial expressions are one of the prominent ways to correctly infer an individual's mood [33] and thus can fulfill the above vision if the smartphone can monitor the temporal changes of its user's facial expressions. Interestingly, there have been decades of research on inferring facial expressions from video or image-based data [3,34,36,51]; however, these works are not suitable to fulfill the above vision of developing a pervasive smartphone application because of the following reasons. Firstly, image and video processing is computationally heavy and consumes a significant amount of system resources and energy.…”