Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition

Abstract

Hand gesture recognition methods play an important role in human-computer interaction. Among these methods are skeleton-based recognition techniques that seem to be promising. In literature, several methods have been proposed to recognize hand gestures with skeletons. One problem with these methods is that they consider little the connectivity between the joints of a skeleton, constructing simple graphs for skeleton connectivity. Observing this, we built a new model of hand skeletons by adding three types of edges in the graph to finely describe the linkage action of joints. Then, an end-to-end deep neural network, hand gesture graph convolutional network, is presented in which the convolution is conducted only on linked skeleton joints. Since the training dataset is relatively small, this work proposes expanding the coordinate dimensionality so as to let models learn more semantic features. Furthermore, relative coordinates are employed to help hand gesture graph convolutional network learn the feature representation independent of the random starting positions of actions. The proposed method is validated on two challenging datasets, and the experimental results show that it outperforms the state-of-the-art methods. Furthermore, it is relatively lightweight in practice for hand skeleton-based gesture recognition.

Smart healthcare solutions using the internet of medical things for hand gesture recognition system

“…Li et al [28] modeled spatial-temporal graph convolutional networks to analyze the dynamic hand gesture. A skeleton-based model was proposed by including three types of edges in the graph linked to the action of hand joints.…”

Multi-camera Torso Pose Estimation using Graph Neural Networks

“…An example of the extraction of information of body parts by a GNN can be seen in [22], where the features of the hand are encoded in a graph of different points and a GCN yields the hand gesture. Similarly, [23] uses the coordinates of the human skeleton joints as the input of a GCN to recognise actions performed on videos.…”

Learning Three Dimensional Tennis Shots Using Graph Convolutional Networks

“…That is why in the research GNNs are connected with convolutional networks (GCNs). Spatial-Temporal Graph Neural Networks (ST-GNNs) are often used in image and video processing, especially for identifying human action patterns [5][6][7][8][9], but also for image classification [10] or semi-supervised learning [11]. This method is of great interest, because it is able to perform automatic analysis based on spatial configuration and by temporal dynamics.…”

“…al. [8] propose a new model for the hand gesture graph convolutional network (HG-GSN) based on the previous work. The skeleton-based action recognition based on a two-stream adaptive graph convolutional network is described in [25].…”