A Unified Framework for Multi-Modal Isolated Gesture Recognition

Abstract

In this article, we focus on isolated gesture recognition and explore different modalities by involving RGB stream, depth stream, and saliency stream for inspection. Our goal is to push the boundary of this realm even further by proposing a unified framework that exploits the advantages of multi-modality fusion. Specifically, a spatial-temporal network architecture based on consensus-voting has been proposed to explicitly model the long-term structure of the video sequence and to reduce estimation variance when confronted with comprehensive inter-class variations. In addition, a three-dimensional depth-saliency convolutional network is aggregated in parallel to capture subtle motion characteristics. Extensive experiments are done to analyze the performance of each component and our proposed approach achieves the best results on two public benchmarks, ChaLearn IsoGD and RGBD-HuDaAct, outperforming the closest competitor by a margin of over 10% and 15%, respectively. Our project and codes will be released at https://davidsonic.github.io/index/acm_tomm_2017.html.

Results and Analysis of ChaLearn LAP Multi-modal Isolated and Continuous Gesture Recognition, and Real Versus Fake Expressed Emotions Challenges

“…Compared with the performances of the first round, the best recognition rate r obtained in round 2 improved considerably (from 56.90% to 67.71% on the test set). We notice that the new baseline [10] also achieved the second best performance. This baseline uses multiple modalities (RGB, depth, optical flow and saliency streams) and a spatio-temporal network architecture, with a consensus-voting strategy (see [10] for details).…”

“…We notice that the new baseline [10] also achieved the second best performance. This baseline uses multiple modalities (RGB, depth, optical flow and saliency streams) and a spatio-temporal network architecture, with a consensus-voting strategy (see [10] for details). Table 2 shows a brief summary of each participants/teams' methodology.…”

“…In the following, we first report the details of isolated and continuous gesture challenges respectively, and then give a brief conclusion for each track. Table 1 shows the final ranking of the isolated gesture recognition challenge, where results of five teams/participants and a new baseline [10] have been reported. For completeness, we report in that table the performances obtained in rounds 1 & 2.…”

Particle Filter Based Probabilistic Forced Alignment for Continuous Gesture Recognition

“…For isolated recognition tasks such as Isolated Gesture [10] and Action Recognition [23], most datasets provide Instance Level Annotations that is a single label for each video clip which does not contain any temporal localisation. To train deep networks using instance level annotations, researchers [11,21,27,28] frequently assign the provided instance labels to all time steps and train neural networks using Cross Entropy Loss [17].…”

“…To train deep networks using instance level annotations, researchers [11,21,27,28] frequently assign the provided instance labels to all time steps and train neural networks using Cross Entropy Loss [17]. However, identifying every part of a sequence with the same label can cause class ambiguity as different stages of a sequence can have different spatio-temporal features.…”