A Novel Geometric Framework on Gram Matrix Trajectories for Human Behavior Understanding

Abstract

In this paper, we propose a novel space-time geometric representation of human landmark configurations and derive tools for comparison and classification. We model the temporal evolution of landmarks as parametrized trajectories on the Riemannian manifold of positive semidefinite matrices of fixed-rank. Our representation has the benefit to bring naturally a second desirable quantity when comparing shapes -the spatial covariance -in addition to the conventional affine-shape representation. We derived then geometric and computational tools for rate-invariant analysis and adaptive re-sampling of trajectories, grounding on the Riemannian geometry of the underlying manifold. Specifically, our approach involves three steps: (1) landmarks are first mapped into the Riemannian manifold of positive semidefinite matrices of fixed-rank to build time-parameterized trajectories; (2) a temporal warping is performed on the trajectories, providing a geometry-aware (dis-)similarity measure between them; (3) finally, a pairwise proximity function SVM is used to classify them, incorporating the (dis-)similarity measure into the kernel function. We show that such representation and metric achieve competitive results in applications as action recognition and emotion recognition from 3D skeletal data, and facial expression recognition from videos. Experiments have been conducted on several publicly available up-to-date benchmarks.In this document, we provide the mathematical preliminaries to our approach. We also give further details on the conducted experiments and the reported results.

Fitting, Comparison, and Alignment of Trajectories on Positive Semi-Definite Matrices with Application to Action Recognition

“…More recently, Kacem et al [18] proposed a geometric approach for modeling and classifying dynamic 2D and 3D landmark sequences based on Gramian matrices derived from the static landmarks. This results in an affineinvariant representation of the data.…”

“…The sequences represented in this manifold can be of different length as the execution rate of the actions can vary from one person to another, meaning that we can not effectively compare them. A common method to do so is to use Dynamic Time Warping (DTW) as proposed in several works [3,18,15]. However, DTW does not define a proper metric and can not be used to derive a valid positive-definite kernel for the classification phase.…”

“…Overall, our approach achieves competitive results with respect to most recent approaches. We directly compare our results with [18] as we work on the same geometric space of S + (d, n) manifold. The main differences between our method and the method in [18] is the use of a different metric and of the Global Alignment Kernel instead of DTW.…”

“…We directly compare our results with [18] as we work on the same geometric space of S + (d, n) manifold. The main differences between our method and the method in [18] is the use of a different metric and of the Global Alignment Kernel instead of DTW. Our metric is simpler that the metric in [18], as we do not have to estimate the parameter k used in Eq.…”

“…This paper lies in the continuity of recent works that model the comparison and classification of temporal sequences of landmarks on the Riemannian manifold of positive-semidefinite matrices. Building on the work [18], our approach involves four different steps: 1) We build a trajectory on the Riemannian manifold from the body skeletons; 2) We apply a curve fitting algorithm on the trajectories to denoise the data points; 3) We perform a temporal alignment using a Global Alignment Kernel, defining a positive-semidefinite kernel; 4) Finally, we use this kernel with a classic SVM to classify the actions. An overview of the full approach is given in Fig.…”