ModDrop: Adaptive Multi-Modal Gesture Recognition

Abstract

Abstract-We present a method for gesture detection and localisation based on multi-scale and multi-modal deep learning. Each visual modality captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at three temporal scales. Key to our technique is a training strategy which exploits: i) careful initialization of individual modalities; and ii) gradual fusion involving random dropping of separate channels (dubbed ModDrop) for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams. Fusing multiple modalities at several spatial and temporal scales leads to a significant increase in recognition rates, allowing the model to compensate for errors of the individual classifiers as well as noise in the separate channels. Futhermore, the proposed ModDrop training technique ensures robustness of the classifier to missing signals in one or several channels to produce meaningful predictions from any number of available modalities. In addition, we demonstrate the applicability of the proposed fusion scheme to modalities of arbitrary nature by experiments on the same dataset augmented with audio.

Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition

“…Inspired by the recent progress in the field of deep learning 2D Convolutional Neural Networks (2D CNNs) have been applied to the Gesture Recognition field in order to extract spatial features [7], [8]. In recent studies, features were either concatenated into fixed sized gesture templates [7] or passed to HMM [13] or Recurrent Neural Networks [8] in order to model the temporal aspects of the gestures.…”

“…In recent studies, features were either concatenated into fixed sized gesture templates [7] or passed to HMM [13] or Recurrent Neural Networks [8] in order to model the temporal aspects of the gestures.…”

“…With the emergence of consumer depth cameras [5], researchers quickly incorporated depth sensors into their systems, as depth simplifies the task of human pose estimation [6]. Many state-of-the-art gesture recognition systems today use depth images as a modality or as a means of preprocessing their data before recognizing gestures [2], [7], [8].…”

“…In recent years, Convolutional Neural Network (CNN) based approaches have achieved state-of-the-art performance in gesture recognition challenges [7], [8]. In [7], Neverova et al proposed a multi-scale and multi-modal deep learning architecture to spot and recognize continuous gestures, and achieved state-of-the-art performance in the ChaLearn 2014 Gesture Recognition challenge [22].…”

“…In recent years, Convolutional Neural Network (CNN) based approaches have achieved state-of-the-art performance in gesture recognition challenges [7], [8]. In [7], Neverova et al proposed a multi-scale and multi-modal deep learning architecture to spot and recognize continuous gestures, and achieved state-of-the-art performance in the ChaLearn 2014 Gesture Recognition challenge [22]. In [8], Pigou et al proposed temporally modeling the spatial features obtained from CNNs by using Recurrent Neural Networks (RNNs) with Long Short-Term Memory units, and shows the benefits of using RNNs over temporal pooling approaches.…”