Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions

Abstract

Recently, the recognition task of spontaneous facial micro-expressions has attracted much attention with its various real-world applications. Plenty of handcrafted or learned features have been employed for a variety of classifiers and achieved promising performances for recognizing micro-expressions. However, the micro-expression recognition is still challenging due to the subtle spatiotemporal changes of micro-expressions. To exploit the merits of deep learning, we propose a novel deep recurrent convolutional networks based micro-expression recognition approach, capturing the spatial-temporal deformations of micro-expression sequence. Specifically, the proposed deep model is constituted of several recurrent convolutional layers for extracting visual features and a classificatory layer for recognition. It is optimized by an end-to-end manner and obviates manual feature design. To handle sequential data, we exploit two types of extending the connectivity of convolutional networks across temporal domain, in which the spatiotemporal deformations are modeled in views of facial appearance and geometry separately. Besides, to overcome the shortcomings of limited and imbalanced training samples, temporal data augmentation strategies as well as a balanced loss are jointly used for our deep network. By performing the experiments on three spontaneous microexpression datasets, we verify the effectiveness of our proposed micro-expression recognition approach compared to the state-ofthe-art methods.

A Spatiotemporal Deep Learning Solution For Automatic Micro-Expressions Recognition From Local Facial Regions

“…An improved solution of Off-ApexNet is STSTNet [8], authors have added to the horizontal and vertical OF the strain OF to get a better result. Xia et al [9] have proposed a Spatiotemporal Recurrent Convolution Network (STRCN). The authors have developed two varieties of the network: STRCN with Appearance based Connectivity (STRCN-A) that consists of a different representation of the image as a vector and so the whole sequence as matrix is provided to a STRCN which is basically a block of recurrent CNN.…”

“…The most commonly used solutions in the state-of-the-art is the hybrid solutions [7,8,9,10]. The primary concept is to use handcrafted solution such as Local Binary Pattern on Three Orthogonal Planes (LBP-TOP) or OF to assist Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) to extract the most significant spatio-temporal features despite the database issues.…”

Multi-Perspective Dynamic Features for Cross-Database Face Presentation Attack Detection

“…As is well known, humans perform many unconscious facial motions, such as blinking and micro-expressions, and these movements can be extracted regardless of the type of camera or imaging environment. Facial motions are controlled by muscles around the eyes, mouth, among others [12], [13]. Although facial contours vary among individuals, the muscular distributions are similar.…”

Feature Selection-Based Hierarchical Deep Network for Image Classification

“…The original hand-craft features are very sensitive to the geometric transformation and quality of the image, which is very unfavorable for large-scale image classification. However, the convolutional layer in deep learning scheme can extract high-level information and achieve good feature expression ability [10]; the nonlinear layer simulation neurons can enhance and suppress the features; the pooling layer can extract the image local information, and ensure the translation rotation invariance, and can also reduce the feature dimension. Through the overall iteration of these operations,…”