Dynamic Micro-Expression Recognition Using Knowledge Distillation

Sun, Bo; Cao, Shixiu; Li, Dongliang; He, Jun; Yu, Lejun

doi:10.1109/taffc.2020.2986962

Cited by 57 publications

(56 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our framework achieves nearly 16.8%, 14.6% and 15.8% increases in accuracy, 9.1%, 12.4% and 33.9% increases in f1 score compared to the best results of handcraft feature methods, i.e., Bi-WOOF [20] on the CASME2, SAMM and SMIC database. Our method also outperforms the action unit assisted method, i.e., Dynamic [32] by 3.0%/3.1% and 0.7%/3.4% on the CASME2 and SMIC database of accuracy/F1. Since Dynamic enforces the features of teacher network ans student network similar by L2-loss, student network would lose its own domain information on micro-expression recognition.…”

Section: 31mentioning

confidence: 82%

“…Due to the lack of large-scale micro-expression datasets, there have been a few works embracing the ideas of using macro-expression images or action unit information to assist micro-expression recognition. Sun et al [32] proposed a knowledge transfer technique that distilled and transferred multi-knowledge from action unit for micro-expression recognition. Sun network on action unit recognition, and transferred it to a student network by penalizing the difference between the features of teacher network and the features of student network.…”

Section: Related Work 21 Micro-expression Recognitionmentioning

confidence: 99%

“…We compare our framework with other related works. These methods are: 1) LBP-TOP [13], LBP-SIP [35], STLBP-IP [8], STCLQP [9], which are LBP based methods 2) HIGO [15], FHOFO [7], Bi-WOOF [20], which are optimal flow based methods, 3) Only-Apex [17], OFF-Apex [5], CNN+LSTM [12], Boost [28], DSSN [11], Shallow [19], Dual [40] Capsule [34], which are deep feature methods, 4) Dynamic [32], which is action unit assisted method. and 5) Neural [21], which is macro-expression assisted method.…”

Section: Comparison With Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Learning from Macro-expression

Xia

Wang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

As one of the most important forms of psychological behaviors, micro-expression can reveal the real emotion. However, the existing labeled micro-expression samples are limited to train a high performance micro-expression classifier. Since micro-expression and macro-expression share some similarities in facial muscle movements and texture changes, in this paper we propose a microexpression recognition framework that leverages macro-expression samples as guidance. Specifically, we first introduce two Expression-Identity Disentangle Network, named MicroNet and MacroNet, as the feature extractor to disentangle expression-related features for micro and macro expression samples. Then MacroNet is fixed and used to guide the fine-tuning of MicroNet from both label and feature space. Adversarial learning strategy and triplet loss are added upon feature level between the MicroNet and MacroNet, so the MicroNet can efficiently capture the shared features of microexpression and macro-expression samples. Loss inequality regularization is imposed to the label space to make the output of Mi-croNet converge to that of MicroNet. Comprehensive experiments on three public spontaneous micro-expression databases, i.e., SMIC, CASME2 and SAMM demonstrate the superiority of the proposed method. CCS CONCEPTS • Human-centered computing → HCI design and evaluation methods.

show abstract

Section: 31mentioning

confidence: 82%

Section: Related Work 21 Micro-expression Recognitionmentioning

confidence: 99%

Section: Comparison With Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning from Macro-expression

Xia

Wang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…Esmaeili et al proposed a feature extractor called Cubic-LBP for FME recognition [18]. Sun et al proposed a knowledge distillation to transfer knowledge from action unit [58].…”

Section: Related Workmentioning

confidence: 99%

Facial Micro-Expression Recognition Using Two-Dimensional Landmark Feature Maps

Choi

Song

2020

IEEE Access

View full text Add to dashboard Cite

Emotion recognition based on facial expressions is very important for effective interaction of humans with artificial intelligence (AI) systems such as social robots. On the other hand, in real environment, it is much harder to recognize facial micro-expressions (FMEs) than facial general-expressions having rich emotions. In this paper, we propose a two-dimensional (2D) landmark feature map for effectively recognizing such FMEs. The proposed 2D landmark feature map (LFM) is obtained by transforming conventional coordinate-based landmark information into 2D image information. LFM is designed to have an advantageous property independent of the intensity of facial expression change. Also, we propose an LFM-based emotion recognition method that is an integrated framework of convolutional neural network (CNN) and long shortterm memory (LSTM). Experimental results show that the proposed method achieves about 71% and 74% in the well-known micro-expression datasets, i.e., SMIC and CASME II, respectively, which outperforms the conventional methods. The performance of the proposed method was also verified through experiments on composite micro-expression dataset, which consists of SMIC, CAMSE II and SAMM, and cross-dataset validation using SMIC and CAMSE II. In addition, we prove that the proposed method is independent of facial expression intensity through an experiment on CK+ dataset. Finally, we demonstrate that the proposed method is valid even for the MAHNOB-HCI and MEVIEW datasets that are produced to monitor actual and wild emotional responses.

show abstract

“…Currently, advances in computer vision-based image classification, such as machine learning and deep learning algorithms, provide unprecedented opportunities for automated ME recognition. In this context, many handcrafted techniques and deep learning methods have been proposed to predict MEs using different feature extraction methods, yielding valuable insights into the progression patterns of ME recognition [10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]. The mainstream feature representation methods for ME recognition are mainly based on the optical flow [13][14][15][16][17], local binary patterns (LBP) [18][19][20][21][22][23], and deep learning techniques [24][25][26][27][28].…”

Section: Introductionmentioning

confidence: 99%

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

Cen

Yan

et al. 2020

Sensors

View full text Add to dashboard Cite

As a spontaneous facial expression, a micro-expression can reveal the psychological responses of human beings. Thus, micro-expression recognition can be widely studied and applied for its potentiality in clinical diagnosis, psychological research, and security. However, micro-expression recognition is a formidable challenge due to the short-lived time frame and low-intensity of the facial actions. In this paper, a sparse spatiotemporal descriptor for micro-expression recognition is developed by using the Enhanced Local Cube Binary Pattern (Enhanced LCBP). The proposed Enhanced LCBP is composed of three complementary binary features containing Spatial Difference Local Cube Binary Patterns (Spatial Difference LCBP), Temporal Direction Local Cube Binary Patterns (Temporal Direction LCBP), and Temporal Gradient Local Cube Binary Patterns (Temporal Gradient LCBP). With the application of Enhanced LCBP, it would no longer be a problem to provide binary features with spatiotemporal domain complementarity to capture subtle facial changes. In addition, due to the redundant information existing among the division grids, which affects the ability of descriptors to distinguish micro-expressions, the Multi-Regional Joint Sparse Learning is designed to perform feature selection for the division grids, thus paying more attention to the critical local regions. Finally, the Multi-kernel Support Vector Machine (SVM) is employed to fuse the selected features for the final classification. The proposed method exhibits great advantage and achieves promising results on four spontaneous micro-expression datasets. Through further observation of parameter evaluation and confusion matrix, the sufficiency and effectiveness of the proposed method are proved.

show abstract

Dynamic Micro-Expression Recognition Using Knowledge Distillation

Cited by 57 publications

References 31 publications

Learning from Macro-expression

Learning from Macro-expression

Facial Micro-Expression Recognition Using Two-Dimensional Landmark Feature Maps

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

Contact Info

Product

Resources

About