Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition

Abstract

Abstract-In this paper, we propose using 3D Convolutional Neural Networks for large scale user-independent continuous gesture recognition. We have trained an end-to-end deep network for continuous gesture recognition (jointly learning both the feature representation and the classifier). The network performs three-dimensional (i.e. space-time) convolutions to extract features related to both the appearance and motion from volumes of color frames. Space-time invariance of the extracted features is encoded via pooling layers. The earlier stages of the network are partially initialized using the work of Tran et al. before being adapted to the task of gesture recognition. An earlier version of the proposed method, which was trained for 11,250 iterations, was submitted to ChaLearn 2016 Continuous Gesture Recognition Challenge and ranked 2nd with the Mean Jaccard Index Score of 0.269235. When the proposed method was further trained for 28,750 iterations, it achieved state-of-the-art performance on the same dataset, yielding a 0.314779 Mean Jaccard Index Score.

Particle Filter Based Probabilistic Forced Alignment for Continuous Gesture Recognition

“…Through our experiments, we have seen the effectiveness of the proposed probabilistic forced alignment approach as it has iteratively improved the recognition performance by 8.1485% relative Mean Jaccard Index Score compared to training with a naive prior distribution. Our method was able the surpass the previous state-of-the-art [3], which also used a 3D-CNN based learning, by obtaining 0.3806 Mean Jaccard Index Score on the validation set. We also participated in the ChaLearn 2017 Continuous Gesture Recognition challenge and were ranked third [30].…”

“…To train our 3D-CNNs we have used Stochastic Gradient Descent [2] with a learning rate of lr = 10 −3 , a momentum of m = 0.99 and a batch size of 45. The networks were initialized with the pre-trained weights provided by Camgoz et al [3] which was the previous state-of-the-art on the ConGD dataset. We re-initialized the last fully connected layer of our 3D-CNNs for each alignment iteration using the Xavier initialization method [16].…”

“…Our method was able to surpass continuous recognition performance of the state-of-the-art [3]. We also participated in the ChaLearn 2017 Continuous Gesture Recognition Challenge, ranking 3rd place [30].…”

Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning

“…The experiment uses scaled uniform distribution [26] for weight initialization. This method is known as Xavier initialization where commonly used in deep learning [27]- [29]. The mini-batch size is set to be 64 running on vary models with 20 epochs per training.…”

Deep Learning for Action and Gesture Recognition in Image Sequences: A Survey

“…In a similar way, Zhu et al (2016a) adopted the same architecture, but this time under a pyramidal for the same problem. In the same line, the work by Camgoz et al (2016) builds an end to end 3D CNN using as basis the model of Tran et al (2015) and applies it to large scale gesture spotting.…”