Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video

Abstract

Recent studies have demonstrated the power of recurrent neural networks for machine translation, image captioning and speech recognition. For the task of capturing temporal structure in video, however, there still remain numerous open research questions. Current research suggests using a simple temporal feature pooling strategy to take into account the temporal aspect of video. We demonstrate that this method is not sufficient for gesture recognition, where temporal information is more discriminative compared to general video classification tasks. We explore deep architectures for gesture recognition in video and propose a new end-to-end trainable neural network architecture incorporating temporal convolutions and bidirectional recurrence. Our main contributions are twofold; first, we show that recurrence is crucial for this task; second, we show that adding temporal convolutions leads to significant improvements. We evaluate the different approaches on the Montalbano gesture recognition dataset, where we achieve state-of-the-art results.

Learning Multimodal Temporal Representation for Dubbing Detection in Broadcast Media

“…As often done in gesture recognition [28] and in NN-based AV speech recognition [23], we consider observations over a short interval (0.2s as in [28,23]) to capture short-term temporal dynamics. Here, a block of 5 visual frames are grouped together (675-dim vector), which corresponds to 20 audio frames (840-dim vector).…”

SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition

“…The emergence of modern deep learning methods [30,13,34] has removed the need for such tailored representations and enabled systems to implicitly learn both the spatial and the temporal features. However, the disadvantage of deep learning is that it can be difficult to encode expert knowledge (such as suitable subunits or intermediate representations).…”

Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition

“…With the emergence of consumer depth cameras [5], researchers quickly incorporated depth sensors into their systems, as depth simplifies the task of human pose estimation [6]. Many state-of-the-art gesture recognition systems today use depth images as a modality or as a means of preprocessing their data before recognizing gestures [2], [7], [8].…”

“…Since the pioneering work of Starner and Pentland [3], Hidden Markov Models have often been used for gesture recognition [13], [14], [15]. Other graphical models such as Hidden Conditional Random Fields [16], Autoregressive Models [17] and Recurrent Neural Networks [8], [18] have also been deployed for the gesture recognition task.…”

“…In recent years, Convolutional Neural Network (CNN) based approaches have achieved state-of-the-art performance in gesture recognition challenges [7], [8]. In [7], Neverova et al proposed a multi-scale and multi-modal deep learning architecture to spot and recognize continuous gestures, and achieved state-of-the-art performance in the ChaLearn 2014 Gesture Recognition challenge [22].…”