Combining 3D Convolutional Neural Networks with Transfer Learning by Supervised Pre-Training for Facial Micro-Expression Recognition

Abstract

Facial micro-expression is momentary and subtle facial reactions, and it is still challenging to automatically recognize facial micro-expression with high accuracy in practical applications. Extracting spatiotemporal features from facial image sequences is essential for facial micro-expression recognition. In this paper, we employed 3D Convolutional Neural Networks (3D-CNNs) for self-learning feature extraction to represent facial micro-expression effectively, since the 3D-CNNs could well extract the spatiotemporal features from facial image sequences. Moreover, transfer learning was utilized to deal with the problem of insufficient samples in the facial micro-expression database. We primarily pretrained the 3D-CNNs on normal facial expression database Oulu-CASIA by supervised learning, then the pre-trained model was effectively transferred to the target domain, which was the facial micro-expression recognition task. The proposed method was evaluated on two available facial micro-expression datasets, i.e. CASME II and SMIC-HS. We obtained the overall accuracy of 97.6% on CASME II, and 97.4% on SMIC, which were 3.4% and 1.6% higher than the 3D-CNNs model without transfer learning, respectively. And the experimental results demonstrated that our method achieved superior performance compared to state-of-the-art methods.

Recognizing Spontaneous Micro-Expression Using a Three-Stream Convolutional Neural Network

“…As shown in Table 3, our TSCNN-I yields the highest recognition accuracy (72.74%) and F 1 -score (0.7236) among other state-of-the-art approaches [16], [22]- [25], [50], [51], [53], [54], [56], [58]- [60], [63]- [69], [72]- [75]. Compared with the best results of other methods (Bi-WOOF+Phase [67], TIM+DCNN+ SVM [75], Dual-Inception Network [25], SSSN [23], DSSN [23], 3D-CNNs [22], OFF-ApexNet [24], and 3D-FCNN [74]), our method yields 4.45%, 6.84%, 6.74%, 9.33%, 9.33%, 6.44%, 4.96%, and 17.25% better recognition accuracy, respectively. 3) Comparison of results using the SAMM database: As shown in Table 4, our TSCNN-I yields a recognition accuracy of 63.53% and an F 1 -score of 0.6065, which are considerably better than the other methods [5], [9], [20], [23], [61], [62], [66].…”

“…Compared with state-of-the-art methods (FMBH [63], OF+CNN [71], ELRCN [19], SSSN [23], DSSN [23], and 3D-CNNs [22]), our method exhibits an improvement of 4.94%, 17.11%, 21.61%, 2.86%, 3.27%, and 8.15% in accuracy, respectively. Thus, our TSCNN-I yielded improved recognition, especially in the absence of index values given by the databases.…”

“…1) Comparison of results using the CASME II database: As shown in Table 2, we report the performance of the TSCNN-I for MER tasks using the CASME II database and compare it with that of other methods [5], [16], [18], [19], [22], [23], [50]- [57], [63], [64], [66], [70], [71], [75] using the LOSO protocol. Our TSCNN-I yielded an accuracy of 74.05% and a mean F 1 -score of 0.7327 using the CASME II database.…”

“…A recent study [21] by Reddy et al proposed two 3D-CNN-based models (MicroExpSTCNN and MicroExpFuseNet) to recognize MEs by extracting both the spatial and temporal information simultaneously by applying a 3D convolution operation to ME videos. Zhi et al [22] similarly suggested 3D convolutional neural networks (3D-CNNs) architecture for self-learning feature extraction to represent facial MEs. Khor et al [23] proposed a lightweight dual-stream shallow network as a pair of truncated CNNs with heterogeneous input features in MER tasks.…”

A New Feature Extraction and Recognition Method for Microexpression Based on Local Non-negative Matrix Factorization

“…However, each dataset has only hundreds of samples, and the number of different microexpressions is seriously unbalanced, which is not sufficient to train a classifier with better generalization ability, especially for deep neural network (DNN). Naturally, researchers hope to train microexpression classifiers by means of numerous macroexpression datasets (Wang et al, 2018 ; Peng et al, 2019 ; Zhi et al, 2019 ). Jia et al ( 2018 ) proposed an MtM algorithm, which uses macroexpression data to generate microexpression samples by constructing corresponding relationship between them.…”