A multi-modal gesture recognition system using audio, video, and skeletal joint data

Abstract

This paper describes the gesture recognition system developed by the Institute for Infocomm Research (I 2 R) for the 2013 ICMI CHALEARN Multi-modal Gesture Recognition Challenge. The proposed system adopts a multi-modal approach for detecting as well as recognizing the gestures. Automated gesture detection is performed using both audio signals and information about hand joints obtained from the Kinect sensor to segment a sample into individual gestures. Once the gestures are detected and segmented, features extracted from three different modalities, namely, audio, 2-dimensional video (RGB), and skeletal joints (Kinect) are used to classify a given sequence of frames into one of the 20 known gestures or an unrecognized gesture. Mel frequency cepstral coefficients (MFCC) are extracted from the audio signals and a Hidden Markov Model (HMM) is used for classification. While Space-Time Interest Points (STIP) are used to represent the RGB modality, a covariance descriptor is extracted from the skeletal joint data. In the case of both RGB and Kinect modalities, Support Vector Machines (SVM) are used for gesture classification. Finally, a fusion scheme is applied to accumulate evidence from all the three modalities and predict the sequence of gestures in each test sample. The proposed gesture recognition system is able to achieve an average edit distance of 0.2074 over the 275 test samples containing 2, 742 unlabeled gestures. While the proposed system is able to recognize the known gestures with high accuracy, most of the errors are caused due to insertion, which occurs when an unrecognized gesture is misclassified as one of the 20 known gestures.

ModDrop: Adaptive Multi-Modal Gesture Recognition

“…In [27], skeletal information was integrated in two ways for extracting HoG features from RGB and depth images: either from global bounding boxes containing a whole body or from regions containing an arm, a torso and a head. Similarly, [28], [29], [30] fused skeletal information with HoG features extracted from either RGB or depth, while [31] proposed a combination of a covariance descriptor representing skeletal joint data with spatio-temporal interest points extracted from RGB augmented with audio.…”

Res3ATN - Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos

“…Supervised learning uses ground truth data for optimizing the training process. Unsupervised learning includes extracting the invariant spatio-temporal features from the videos using independent subspace analysis (ISA), autoencoders, or some other variant network [3,39,35]. Convolutional Restricted Boltzmann Machines (RBMs) have also been used for generating feature representations of the video frames [9].…”

Landmark-based multimodal human action recognition

“…Laptev et al [18] use non-linear SVMs for the task of recognizing daily activities of small temporal length (answer the phone, sit down/up, kiss, hug, get out of car). Similar, authors in [29] use SVMs on temporal and timeweighted variances, and authors in [21] employ SVMs in RGB and Depth data to recover gestures, and then apply a fusion scheme using inferred motion and audio, in a multimodal environment. Authors in [14] have also utilized SVMs for activity feature classification, on joint orientation angles and their forward differences, while view-invariant features (normalized between-joint distances orientations and velocities) have been employed in [28].…”