Covariance Pooling for Facial Expression Recognition

Abstract

Classifying facial expressions into different categories requires capturing regional distortions of facial landmarks. We believe that second-order statistics such as covariance is better able to capture such distortions in regional facial features. In this work, we explore the benefits of using a manifold network structure for covariance pooling to improve facial expression recognition. In particular, we first employ such kind of manifold networks in conjunction with traditional convolutional networks for spatial pooling within individual image feature maps in an end-to-end deep learning manner. By doing so, we are able to achieve a recognition accuracy of 58.14% on the validation set of Static Facial Expressions in the Wild (SFEW 2.0) and 87.0% on the validation set of Real-World Affective Faces (RAF) Database 1 . Both of these results are the best results we are aware of. Besides, we leverage covariance pooling to capture the temporal evolution of per-frame features for video-based facial expression recognition. Our reported results demonstrate the advantage of pooling image-set features temporally by stacking the designed manifold network of covariance pooling on top of convolutional network layers. arXiv:1805.04855v1 [cs.CV] 13 May 2018Accordingly, the core techniques of the two proposed models are spatial/temporal covariance pooling and the manifold network for learning the second-order features deeply. In the following we will introduce the two crucial techniques.

Automatic Analysis of Facial Expressions Based on Deep Covariance Trajectories

“…Accuracy # classes D/S Jung et al [18] 74.17 6 Dynamic Kacem et al [20] 83. 13 6 Dynamic Liu et al [41] 74.59 6 Dynamic Guo et al [47] 75.52 6 Dynamic Cai et al [42] 77. 29 6 Static Ding et al [4] 82.…”

“…The two networks are then combined using a joint fine-tuning method. Acharya et al [13] have extended their static approach discussed before to dynamic facial expression recognition. They considered the temporal evolution of per-frame features by leveraging covariance pooling.…”

Few-Shot Learning With Localization in Realistic Settings

“…Recent work has shown that similar techniques [7,22,27], learnable generalizations of these techniques [8,48], and efficient approximations to these techniques [16,20] also improve the performance of convolutional networks. The improvement is especially large in fine-grained classification settings, such as facial recognition [4,9,27]. However, using the resulting expanded feature space requires parameter- heavy models, even in the few-shot setting [49].…”

“…For hard classification problems, methods such as bilinear pooling [27], fisher vectors [34] and others [4,16,22] can be used to expand the feature space and increase expressive power. Unfortunately, a traditional learning framework uses these expanded representations as input to linear classifiers, fully-connected softmax layers, or multilayer networks [9,20,27,49], dramatically increasing parameters and making the model prone to catastrophic overfitting.…”

“…(1), which are averaged to produce foreground and background feature vectors (2). Pixel features on new feature maps (3) are classified as foreground or background based on distance from those vectors (4). The predicted mask separates foreground and background regions (5), which are average pooled independently and concatenated (6).…”