Hand gesture recognition with 3D convolutional neural networks

Molchanov, Pavlo; Gupta, Shalini; Kim, Kihwan; Kautz, Jan

doi:10.1109/cvprw.2015.7301342

Cited by 410 publications

(228 citation statements)

References 14 publications

Supporting

Mentioning

210

Contrasting

Unclassified

Order By: Relevance

“…a MacBook Pro core-i7 computer by only using CPU and 16GB memory. Table 4 demonstrates that C3D [18] and R3DCNN [5] rely heavily on computational resources, which can only run at 1fps.…”

Section: Evaluation On the Public Datasets For Hgrmentioning

confidence: 99%

LE-HGR: A Lightweight and Efficient RGB-Based Online Gesture Recognition Network for Embedded AR Devices

Xie

Wang

Shao

et al. 2019

2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

View full text Add to dashboard Cite

Figure 1: LE-HGR Framework. The overall framework of the proposed gesture recognition algorithm. The framework contains four sub-modules: Hand candidate detector, multi-task CNN, hand trace mapping, and trace sequence network. ABSTRACTOnline hand gesture recognition (HGR) techniques are essential in augmented reality (AR) applications for enabling natural humanto-computer interaction and communication. In recent years, the consumer market for low-cost AR devices has been rapidly growing, while the technology maturity in this domain is still limited. Those devices are typical of low prices, limited memory, and resourceconstrained computational units, which makes online HGR a challenging problem. To tackle this problem, we propose a lightweight and computationally efficient HGR framework, namely LE-HGR, to enable real-time gesture recognition on embedded devices with low computing power. We also show that the proposed method is of high accuracy and robustness, which is able to reach high-end performance in a variety of complicated interaction environments.To achieve our goal, we first propose a cascaded multi-task convolutional neural network (CNN) to simultaneously predict probabilities of hand detection and regress hand keypoint locations online. We show that, with the proposed cascaded architecture design, false-positive estimates can be largely eliminated. Additionally, an associated mapping approach is introduced to track the hand trace via the predicted locations, which addresses the interference of multi-handedness. Subsequently, we propose a trace sequence neural network (TraceSeqNN) to recognize the hand gesture by exploiting the motion features of the tracked trace. Finally, we provide a variety of experimental results to show that the proposed framework is able to achieve state-of-the-art accuracy with significantly reduced computational cost, which are the key properties for enabling real-time applications in low-cost commercial devices such as mobile devices and AR/VR headsets. * These authors contributed equally to this work.

show abstract

“…a MacBook Pro core-i7 computer by only using CPU and 16GB memory. Table 4 demonstrates that C3D [18] and R3DCNN [5] rely heavily on computational resources, which can only run at 1fps.…”

Section: Evaluation On the Public Datasets For Hgrmentioning

confidence: 99%

LE-HGR: A Lightweight and Efficient RGB-Based Online Gesture Recognition Network for Embedded AR Devices

Xie

Wang

Shao

et al. 2019

2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

View full text Add to dashboard Cite

show abstract

“…CNNs have been successfully applied to a diverse set of non-imaging domains, including natural language processing [30], bird song segmentation [31], and EEG recordings [32]. Perhaps most clearly mirroring our challenge of motion classification, CNNs have performed well in the classification of video recordings [33], [34], [35], [36].…”

Section: A Related Workmentioning

confidence: 99%

Deep convolutional and recurrent neural networks for cell motility discrimination and prediction

Kimmel

Brack

Marshall

2017

Preprint

View full text Add to dashboard Cite

Cells in culture display diverse motility behaviors that may reflect differences in cell state and function, providing motivation to discriminate between different motility behaviors. Current methods to do so rely upon manual feature engineering. However, the types of features necessary to distinguish between motility behaviors can vary greatly depending on the biological context, and it is not always clear which features may be most predictive in each setting for distinguishing particular cell types or disease states. Convolutional neural networks (CNNs) are machine learning models allowing for relevant features to be learned directly from spatial data. Similarly, recurrent neural networks (RNNs) are a class of models capable of learning long term temporal dependencies. Given that cell motility is inherently spacio-temporal data, we present an approach utilizing both convolutional and long-short-term memory (LSTM) recurrent neural network units to analyze cell motility data. These RNN models provide accurate classification of simulated motility and experimentally measured motility from multiple cell types, comparable to results achieved with hand-engineered features. The variety of cell motility differences we can detect suggests that the algorithm is generally applicable to additional cell types not analyzed here. RNN autoencoders based on the same architecture are capable of learning motility features in an unsupervised manner and capturing variation between myogenic cells in the latent space. Adapting these RNN models to motility prediction, RNNs are capable of predicting muscle stem cell motility from past tracking data with performance superior to standard motion prediction models. This advance in cell motility prediction may be of practical utility in cell tracking applications.

show abstract

“…In a more recent study, 3D Convolutional Neural Networks (3D CNNs) were proposed to recognize isolated gestures [23]. Depth and intensity information were combined into a single image and these in turn combined to form gesture volumes.…”

Section: D Convolutional Neural Networkmentioning

confidence: 99%

“…In [8], Pigou et al proposed temporally modeling the spatial features obtained from CNNs by using Recurrent Neural Networks (RNNs) with Long Short-Term Memory units, and shows the benefits of using RNNs over temporal pooling approaches. In a recent study, 3D Convolutional Neural Networks were proposed for the isolated hand gesture recognition task using depth and intensity modalities in automotive interfaces and reported a better recognition performance than using HOG descriptors [23].…”

Section: Introductionmentioning

confidence: 99%

Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition

Camgöz

Hadfield

Koller

et al. 2016

2016 23rd International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Abstract-In this paper, we propose using 3D Convolutional Neural Networks for large scale user-independent continuous gesture recognition. We have trained an end-to-end deep network for continuous gesture recognition (jointly learning both the feature representation and the classifier). The network performs three-dimensional (i.e. space-time) convolutions to extract features related to both the appearance and motion from volumes of color frames. Space-time invariance of the extracted features is encoded via pooling layers. The earlier stages of the network are partially initialized using the work of Tran et al. before being adapted to the task of gesture recognition. An earlier version of the proposed method, which was trained for 11,250 iterations, was submitted to ChaLearn 2016 Continuous Gesture Recognition Challenge and ranked 2nd with the Mean Jaccard Index Score of 0.269235. When the proposed method was further trained for 28,750 iterations, it achieved state-of-the-art performance on the same dataset, yielding a 0.314779 Mean Jaccard Index Score.

show abstract

Hand gesture recognition with 3D convolutional neural networks

Cited by 410 publications

References 14 publications

LE-HGR: A Lightweight and Efficient RGB-Based Online Gesture Recognition Network for Embedded AR Devices

LE-HGR: A Lightweight and Efficient RGB-Based Online Gesture Recognition Network for Embedded AR Devices

Deep convolutional and recurrent neural networks for cell motility discrimination and prediction

Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition

Contact Info

Product

Resources

About