A Novel Space-Time Representation on the Positive Semidefinite Cone for Facial Expression Recognition

Abstract

In this paper, we study the problem of facial expression recognition using a novel space-time geometric representation. We describe the temporal evolution of facial landmarks as parametrized trajectories on the Riemannian manifold of positive semidefinite matrices of fixed-rank. Our representation has the advantage to bring naturally a second desirable quantity when comparing shapes -the spatial covariance -in addition to the conventional affineshape representation. We derive then geometric and computational tools for rate-invariant analysis and adaptive re-sampling of trajectories, grounding on the Riemannian geometry of the manifold. Specifically, our approach involves three steps: 1) facial landmarks are first mapped into the Riemannian manifold of positive semidefinite matrices of rank 2, to build time-parameterized trajectories; 2) a temporal alignment is performed on the trajectories, providing a geometry-aware (dis-)similarity measure between them; 3) finally, pairwise proximity function SVM (ppfSVM) is used to classify them, incorporating the latter (dis-)similarity measure into the kernel function. We show the effectiveness of the proposed approach on four publicly available benchmarks (CK+, MMI, Oulu-CASIA, and AFEW). The results of the proposed approach are comparable to or better than the state-of-the-art methods when involving only facial landmarks.

Sparse Coding of Shape Trajectories for Facial Expression and Action Recognition

“…From Fig. 4 and the confusion matrix in Table 2, we can observe that the two expressions: happiness and surprise are well recognized in the two datasets while the main confusions happened in the two expressions: fear and sadness, conforming to state-of-the-art results [35], [37]. Besides, we highlight the superiority of extrinsic SCDL compared to intrinsic SCDL.…”

“…Similarly, on Oulu-CASIA, our best result is lower than DTAGN and higher than DTGN. On the other hand, the method of [37] achieved a better performance on both datasets compared to our method. Comparing the confusion matrices, the same method seems to better recognize the sadness expression while our method is clearly more efficient in recognizing the contempt expression.…”

Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

“…AdaLBP (Zhao et al 2011) 73.54 Atlases (Guo, Zhao, and Pietikinen 2012) 75.52 ExpLet (Liu et al 2016) 76.65 Dis-ExpLet (Liu et al 2014) 79.00 Lomo (Sikka, Sharma, and Bartlett 2016) 82.10 DTAGN (Jung et al 2015) 81 Table 2 shows a comparison with previously recorded methods(including spatio-temporal and appearance based methods) (Elaiwat, Bennamoun, and Boussaid 2016;Kacem et al 2017;Hu et al 2017;Vielzeuf, Pateux, and Jurie 2017;Yao et al 2016;Ebrahimi Kahou et al 2015). On the other hand, no previously recorded methods are available on the KAIST Face MPMI datasets.…”

Automatic Analysis of Facial Expressions Based on Deep Covariance Trajectories

“…This issue necessitates the use of an algorithm based generally on dynamic programming to align different trajectories. Several works including [5], [6], [20] used DTW to align trajectories in a Riemannian manifold; however, this algorithm does not define a proper metric, which is indeed required in the classification phase to define a valid positive-definite kernel. As alternative solution, different works [6], [20], [23] proposed to ignore this constraint by using a variant of SVM with an arbitrary kernel without any restrictions on the kernel function.…”