Enriched Long-Term Recurrent Convolutional Network for Facial Micro-Expression Recognition

Abstract

Facial micro-expression (ME) recognition has posed a huge challenge to researchers for its subtlety in motion and limited databases. Recently, handcrafted techniques have achieved superior performance in micro-expression recognition but at the cost of domain specificity and cumbersome parametric tunings. In this paper, we propose an Enriched Long-term Recurrent Convolutional Network (ELRCN) that first encodes each micro-expression frame into a feature vector through CNN module(s), then predicts the micro-expression by passing the feature vector through a Long Short-term Memory (LSTM) module. The framework contains two different network variants: (1) Channel-wise stacking of input data for spatial enrichment, (2) Feature-wise stacking of features for temporal enrichment. We demonstrate that the proposed approach is able to achieve reasonably good performance, without data augmentation. In addition, we also present ablation studies conducted on the framework and visualizations of what CNN "sees" when predicting the micro-expression classes.

A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges

“…Two protocols – Hold-out Database Evaluation (HDE) and Composite Database Evaluation (CDE), were proposed in the challenge, using the CASME II and SAMM databases. The reported performances (Khor et al, 2018; Merghani et al, 2018; Peng et al, 2018) were poorer than most other works that apply only to single databases, indicating that future methods need to be more robust across domains.…”

Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition

“…Besides transfer learning, [57] adopt two domain adaptation techniques including adversarial training and expression magnification obtain the best results on the full composite database in MEGC2019. Other methods [58]- [63] also show promising performance in cross-database challenges.…”

A Spatiotemporal Deep Learning Solution For Automatic Micro-Expressions Recognition From Local Facial Regions

“…Thanks to the recent developments in hardware and software technologies those kind of solution become possible. Khor et al [10] have proposed an Enriched Long-term Recurrent Convolutional Network (ELRCN). First, several types of OF are calculated {Horizontal, Vertical, Magnitude and Strain}.…”

“…The most commonly used solutions in the state-of-the-art is the hybrid solutions [7,8,9,10]. The primary concept is to use handcrafted solution such as Local Binary Pattern on Three Orthogonal Planes (LBP-TOP) or OF to assist Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) to extract the most significant spatio-temporal features despite the database issues.…”