Hidden States Exploration for 3D Skeleton-Based Gesture Recognition

Abstract

3D skeletal data has recently attracted wide attention in human behavior analysis for its robustness to variant scenes, while accurate gesture recognition is still challenging. The main reason lies in the high intra-class variance caused by temporal dynamics. A solution is resorting to the generative models, such as the hidden Markov model (H-MM). However, existing methods commonly assume fixed anchors for each hidden state, which is hard to depict the explicit temporal structure of gestures. Based on the observation that a gesture is a time series with distinctly defined phases, we propose a new formulation to build temporal compositions of gestures by the low-rank matrix decomposition. The only assumption is that the gesture's "hold" phases with static poses are linearly correlated among each other. As such, a gesture sequence could be segmented into temporal states with semantically meaningful and discriminative concepts. Furthermore, different to traditional H-MMs which tend to use specific distance metric for clustering and ignore the temporal contextual information when estimating the emission probability, the Long Short-Term Memory (LSTM) is utilized to learn probability distributions over states of HMM. The proposed method is validated on two challenging datasets. Experiments demonstrate that our approach can effectively work on a wide range of gestures and actions, and achieve state-of-the-art performance.

Literature review of vision‐based dynamic gesture recognition using deep learning techniques

“…Liu et al 79 suggest a new framework for recognition of gestures in skeleton sequences, which will absorb the benefits of HMM and LSTM. Hidden states with short time length are given as input to the LSTM rather than model the whole gesture sequences within the LSTM.…”

Approach for Improving User Interface Based on Gesture Recognition

“…On the other hand, a machine learning approach entails creating a mapping feature from gestural data to gestureclasses. To create the functional mapping from gestural instances to gesture-classes, many machine learning techniques have been used [6,7]. Since the classes are identified ahead of time, gesture recognition is achieved by supervised learning, in which a supervisor modifies the training data collection, defining gestural instances and the corresponding class details for each gesture instance.…”

Reconhecimento online de gestos dinâmicos para ambientes interacionais multicâmeras

“…Mesmo com a possibilidade de utilização de esqueletos 3D, alguns trabalhos utilizam modelos baseados em sequências fixas de imagens (Escobedo-Cardenas and Camara-Chavez, 2015;Liu et al, 2019), o que acaba gerando uma dependência de como os gestos devem ser executados, e por conseguinte limita a sua naturalidade de execução. Um problema similar ocorre com as abordagens baseadas em regras (Santos et al, 2015), que apresentam uma forte dependência da percepção do projetista.…”