MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Face Images

Abstract

This paper is aimed at creating extremely small and fast convolutional neural networks (CNN) for the problem of facial expression recognition (FER) from frontal face images. To this end, we employed the popular knowledge distillation (KD) method and identified two major shortcomings with its use: 1) a fine-grained grid search is needed for tuning the temperature hyperparameter and 2) to find the optimal size-accuracy balance, one needs to search for the final network size (i.e. the compression rate). On the other hand, KD is proved to be useful for model compression for the FER problem, and we discovered that its effects gets more and more significant with the decreasing model size. In addition, during the search for a network architecture, we hypothesized that translation invariance achieved using max-pooling layers would not be useful for the FER problem as the expressions are sensitive to small, pixel-wise changes around the eye and the mouth. However, we have found an intriguing improvement on generalization when max-pooling is used. Experiments are made on two widelyused FER datasets, CK+ and Oulu-CASIA. Our smallest model (MicroExpNet), obtained using knowledge distillation, is less than 1MB in size and works at 1851 frames per second on an Intel i7 CPU. Despite being less accurate than the state-of-the-art, MicroExpNet still provides significant insights on the creation of a micro architecture for the FER problem.

REsCUE

“…Facial Expression. The proposed system uses MicroExpNet [14] to extract the facial expressions of the coachees. This is a small and fast convolutional neural network designed for facial expression recognition, which is obtained by distilling a heavy and accurate neural network.…”

“…This is a small and fast convolutional neural network designed for facial expression recognition, which is obtained by distilling a heavy and accurate neural network. Çugu et al [14] reported that the network achieved over 95.0% classification accuracy for the eight expressions of "neutral, " "anger, " "contempt, " "disgust, " "fear, " "happy, " "sadness, " and "surprise" under the real-time conditions. Therefore, we decided to use this network to meet requirement (2).…”

Self-Difference Convolutional Neural Network for Facial Expression Recognition

“…Our SD-CNN method is compared with the recent state-of-the-art FER methods, including DeRL [ 12 ], FN2EN [ 18 ], FMPN [ 28 ], VGG-face [ 8 ], MicroExpNet [ 34 ], GoogLeNet [ 17 ], MultiAttention [ 24 ], DSAE [ 26 ], GCNet [ 40 ], DynamicMTL [ 41 ], IA-gen [ 20 ], CompactCNN [ 30 ], DTAGN(Joint) [ 29 ], CPPN [ 27 ], DPND [ 10 ], PPDN [ 11 ], and FAN [ 31 ]. Table 2 reports our experimental results and shows the comparisons with these methods.…”

“…Our SD-CNN is also compared with the recent state-of-the-art methods, including FN2EN [ 18 ], DeRL [ 12 ], GoogLeNet (fine-tuned) [ 17 ], VGG-face (fine-tuned) [ 8 ], GCNet [ 40 ], DynamicMTL [ 41 ], MicroExpNet [ 34 ], MultiAttention [ 24 ], IA-gen [ 20 ], PPDN [ 11 ], DPND [ 10 ], DTAGN(Joint) [ 29 ], and CompactCNN [ 30 ], on the Oulu-CASIA dataset. As shown in Table 3 , our method achieves the highest accuracy of 91.3 %, which outperforms the state-of-the-art static image-based method (i.e., the DynamicMTL [ 41 ]) by 1.7% and also suppresses the state-of-the-art sequence-based method (i.e., the CompactCNN [ 30 ]) by 2.7%.…”

“…According to the Facial Action Coding System (FACS) [ 33 ], each of the six facial expressions can be represented by a vector of empirical action unit intensities that: . In our implementation, empirical action unit intensities are calculated from the EmotioNet dataset [ 34 ]. To make the generated facial expressions more diverse, we sample an action unit vector for each subject from the empirical action unit intensities using the Gaussian distribution.…”