Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods

Abstract

Abstract-Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i) We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii) We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii) We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges

“…Thus, feature-based approach are usually applied when the local structures are more significant than the information carried by the image intensities. In some ME works (Shreve et al, 2011; Moilanen et al, 2014; Li et al, 2017), the centroid of the two detected eyes are selected as the distinctive point (also called control points) and exploited for face registration by using affine transform or non-reflective similarity transform. The consequence of such simplicity entails their inability to handle deformations locally.…”

“…The consequence of such simplicity entails their inability to handle deformations locally. A number of works (Li et al, 2017; Xu et al, 2017) employed Local Weighted Mean (LWM) (Goshtasby, 1988) which seeks to find a 2-D transformation matrix using 68 facial landmark points of a model face (typically from the first frame). In another work by Xia et al (2016), Procrustes analysis is applied to align the detected landmark points in frames.…”

“…Instead of using mapping functions to map the sensed images to the reference images, a few studies (Shreve et al, 2011; Moilanen et al, 2014; Li et al, 2017) correct the mis-alignment by rotating the faces according to the angle between the pair of lines that join the centroids of the two detected eyes to the horizontal line. In this mechanism, errors can creep in if the face contours of the sensed and reference face images are not consistent with one another, or that the subject's face is not entirely symmetrical to begin with.…”

“…Another popular facial segmentation method is splitting the face into a specific number ( m × n ) of blocks (Moilanen et al, 2014; Davison et al, 2015, 2016a; Wang et al, 2016a; Li et al, 2017). In the blocking representation, the motion changes in each block could by observed and analysis independently.…”

Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets

“…Compare to traditional hand-engineered features, such as Local Binary Pattern (LBP) [31] and Histogram of Gradients (HoG) [1,16]; a deep convolutional neural network consists of multiple layers which can automatically learn hierarchies visual features directly from the raw image pixels. In the research [12], Kim et al introduced deep learning features for micro-expression recognition.…”