A Deep Learning Framework for Recognizing Both Static and Dynamic Gestures

Abstract

Intuitive user interfaces are indispensable to interact with the human centric smart environments. In this paper, we propose a unified framework that recognizes both static and dynamic gestures, using simple RGB vision (without depth sensing). This feature makes it suitable for inexpensive human-robot interaction in social or industrial settings. We employ a pose-driven spatial attention strategy, which guides our proposed Static and Dynamic gestures Network—StaDNet. From the image of the human upper body, we estimate his/her depth, along with the region-of-interest around his/her hands. The Convolutional Neural Network (CNN) in StaDNet is fine-tuned on a background-substituted hand gestures dataset. It is utilized to detect 10 static gestures for each hand as well as to obtain the hand image-embeddings. These are subsequently fused with the augmented pose vector and then passed to the stacked Long Short-Term Memory blocks. Thus, human-centred frame-wise information from the augmented pose vector and from the left/right hands image-embeddings are aggregated in time to predict the dynamic gestures of the performing person. In a number of experiments, we show that the proposed approach surpasses the state-of-the-art results on the large-scale Chalearn 2016 dataset. Moreover, we transfer the knowledge learned through the proposed methodology to the Praxis gestures dataset, and the obtained results also outscore the state-of-the-art on this dataset.

Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

“…One approach that attempts to integrate static and dynamic recognition is the architecture of Mazhar et al [13]. In their CNNLSTM-based framework called StaDNet, they propose an architecture consisting of two Inception V3 CNNs.…”

“…This issue is also prominent in recent state-of-the-art approaches. Many of the modern gesture recognition techniques, such as 3DCNN [23], ResNet [17] and Inception V3 [13] require advanced transform learning techniques and relatively long training times. In contrast, our system addresses this problem by using a simple additional static channel.…”

“…Thus, it has a powerful false-positive limiting characteristic, making it a viable asset in critical scenario applications. One example of safety is shown in the literature in which the human operator has control over a robotic arm [13]. The study addresses a physical safety scenario where the communication is carried out through gestures.…”

“…Our system accomplishes that by extracting the pose of the hand actively used when making the gesture. That makes our system unique to other studies that mirror all videos of left-handed subjects [23] or have a dedicated network for each hand [13]. Thus, our approach facilitates higher flexibility, which leads to less restrictive and guided HRI scenarios.…”

Convolutional neural networks framework for human hand gesture recognition

“…In recent years, the convolutional neural networks (CNN) overtake the complex pre-processing of images and assist in classifying and recognizing images, therefore, it is extensively utilized when handling images. Numerous researchers have started to implement CNN for recognizing human gestures and achieved good results [14]- [17]. In this paper, the proposed CNN framework works on recognizing static hand gestures for obtaining effective and accurate results.…”