Multi-scale fusion visual attention network for facial micro-expression recognition

Abstract

IntroductionMicro-expressions are facial muscle movements that hide genuine emotions. In response to the challenge of micro-expression low-intensity, recent studies have attempted to locate localized areas of facial muscle movement. However, this ignores the feature redundancy caused by the inaccurate locating of the regions of interest.MethodsThis paper proposes a novel multi-scale fusion visual attention network (MFVAN), which learns multi-scale local attention weights to mask regions of redundancy features. Specifically, this model extracts the multi-scale features of the apex frame in the micro-expression video clips by convolutional neural networks. The attention mechanism focuses on the weights of local region features in the multi-scale feature maps. Then, we mask operate redundancy regions in multi-scale features and fuse local features with high attention weights for micro-expression recognition. The self-supervision and transfer learning reduce the influence of individual identity attributes and increase the robustness of multi-scale feature maps. Finally, the multi-scale classification loss, mask loss, and removing individual identity attributes loss joint to optimize the model.ResultsThe proposed MFVAN method is evaluated on SMIC, CASME II, SAMM, and 3DB-Combined datasets that achieve state-of-the-art performance. The experimental results show that focusing on local at the multi-scale contributes to micro-expression recognition.DiscussionThis paper proposed MFVAN model is the first to combine image generation with visual attention mechanisms to solve the combination challenge problem of individual identity attribute interference and low-intensity facial muscle movements. Meanwhile, the MFVAN model reveal the impact of individual attributes on the localization of local ROIs. The experimental results show that a multi-scale fusion visual attention network contributes to micro-expression recognition.

MCNet: meta-clustering learning network for micro-expression recognition

“…While the MCNet method on SMIC dataset was less effective than the optimal MFVAN method, 33 it achieved significantly higher recognition scores than MFVAN method on the CASME II dataset. Additionally, on the CASME dataset, the SVM+LCBP-STGCN method 36 outperformed the proposed method in accuracy and F1-score by 10.99% and 0.07, respectively.…”

“…Specifically, the prediction accuracy reached 65.63%, and the F1-score reached 0.65 on the SMIC dataset, showing 1.03% improvement over the accuracy (64.60%) achieved by the latest dual-ATME method. 33 For the CASME dataset, the prediction accuracy was 70.27%, and the F1-score was 0.7, indicating an improvement of 0.68% in accuracy over the Meta-MMFNet method 24 and an increase of 0.1 in F1-score over the LGCconD method. 47 Moreover, on the CASME II dataset, the prediction accuracy achieved 81.63%, and the F1-score achieved 0.81, which represents an accuracy improvement of 0.68% over the performance of the Meta-MMFNet.…”

“…In addition to handling discriminative representations of micro-expression sequences from a dynamic feature perspective, deep learning methods based on attention mechanisms have also demonstrated success. [31][32][33][34] For instance, Zhou et al 32 proposed the dual-branch attention network (dual-ATME), comprising hand-crafted attention region selection (HARS) and automated attention region selection (AARS). The HARS manually extracted features from the ROI using prior knowledge, while AARS automatically extracted hidden information from the sequence based on attention mechanisms.…”

Graph Convolutional Neural Networks for Micro-Expression Recognition—Fusion of Facial Action Units for Optical Flow Extraction

“…Prototype network is a method of less sample learning. Sample less learning is the process of allowing a model to learn a new task or category with only a few samples [21][22][23]. Prototype networks aim to classify or compare between categories by learning to compute a prototype for each category.…”