Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition

Abstract

In the recent year, state-of-the-art for facial microexpression recognition have been significantly advanced by deep neural networks. The robustness of deep learning has yielded promising performance beyond that of traditional handcrafted approaches. Most works in literature emphasized on increasing the depth of networks and employing highly complex objective functions to learn more features. In this paper, we design a Shallow Triple Stream Three-dimensional CNN (STSTNet) that is computationally light whilst capable of extracting discriminative high level features and details of micro-expressions. The network learns from three optical flow features (i.e., optical strain, horizontal and vertical optical flow fields) computed based on the onset and apex frames of each video. Our experimental results demonstrate the effectiveness of the proposed STSTNet, which obtained an unweighted average recall rate of 0.7605 and unweighted F1-score of 0.7353 on the composite database consisting of 442 samples from the SMIC, CASME II and SAMM databases.

Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System

“…Kim et al [64] proposed to combine CNN with long-term memory (LSTM) and remarkable results are obtained. It has been demonstrated that shallow CNN architectures are practically workable in the ME recognition system [65,66,67,68] . Overall, although the transfer learning [69] and data augmentation [70] are performed, the limited number and imbalance emotion class distribution issues pose a great challenge to develop robust feature extractors.…”

“…Compared to other classifiers such as Random Forest (RF) [72], sparse representation classifier (SRC) [73] and Relaxed K-SVD [74], SVM appears to be more consistent across all the databases with distinct features extracted. Due to the emergence of deep learning, the Softmax classifier, normally served as the final fully connected layer, has been employed in the recent works [64,75,67,66]. Its outstanding discriminative characteristics is beneficial, especially in dealing the high-level features.…”

“…To evaluate the robustness of a feature extractor, [66] combines the video samples from three databases with different nature. Specifically, CASME II, SMIC and SAMM are combined into a composite database, therefore, it contains 442 videos and three emotion classes (i.e., positive, negative and surprise) from 68 participants.…”

A Spatiotemporal Deep Learning Solution For Automatic Micro-Expressions Recognition From Local Facial Regions

“…The classification step is done by an FCL. An improved solution of Off-ApexNet is STSTNet [8], authors have added to the horizontal and vertical OF the strain OF to get a better result. Xia et al [9] have proposed a Spatiotemporal Recurrent Convolution Network (STRCN).…”

“…The most commonly used solutions in the state-of-the-art is the hybrid solutions [7,8,9,10]. The primary concept is to use handcrafted solution such as Local Binary Pattern on Three Orthogonal Planes (LBP-TOP) or OF to assist Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) to extract the most significant spatio-temporal features despite the database issues.…”