Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network

Abstract

Using hand gestures is a natural method of interaction between humans and computers. We use gestures to express meaning and thoughts in our everyday conversations. Gesture-based interfaces are used in many applications in a variety of fields, such as smartphones, televisions (TVs), video gaming, and so on. With advancements in technology, hand gesture recognition is becoming an increasingly promising and attractive technique in human–computer interaction. In this paper, we propose a novel method for fingertip detection and hand gesture recognition in real-time using an RGB-D camera and a 3D convolution neural network (3DCNN). This system can accurately and robustly extract fingertip locations and recognize gestures in real-time. We demonstrate the accurateness and robustness of the interface by evaluating hand gesture recognition across a variety of gestures. In addition, we develop a tool to manipulate computer programs to show the possibility of using hand gesture recognition. The experimental results showed that our system has a high level of accuracy of hand gesture recognition. This is thus considered to be a good approach to a gesture-based interface for human–computer interaction by hand in the future.

Two viewpoints based real‐time recognition for hand gestures

“…Finger count and finger position relative to palm are prominent features in the correct recognition of gestures [27]. Fingers can be detected in many ways such as convexity defects [26], K ‐cosine corner detection [20], and colour depth [10]. The finger count and detection method encompasses calculating the centre of gravity from the segmented hand and thereafter determining the maximum distances to tangent points of fingers [9].…”

“…Artificial neural networks (ANNs) [3, 16] and point pattern matching [8, 17] are some methods used in gesture recognition. Recent deep learning techniques applied in gesture detection and recognition are lightweight CNN [18], LeNet [19], 3DCNN [20], and capsule networks [19] with squash activation function. The nearest‐neighbour algorithm is one of the simplest but effective classifiers which guarantees low error rate [14].…”

3D Skeletal Joints-Based Hand Gesture Spotting and Classification

“…The multi-modal network, which combines a Gaussian-Bernoulli Deep Belief Network (DBN) with skeleton data input and a 3DCNN model with RGB_D data, was effectively utilized for gesture classification by Di et al [7]. Tran et al [22] presented CNN based method using a Kinect Camera for spotting and classification of hand gestures. However, the gesture spotting was done manually from a pre-specified hand shape or finger-tip pattern.…”

A Dynamic Gesture Recognition Interface for Smart Home Control based on Croatian Sign Language

“…To detect the hand gestures, in [38], YCbCr and SkinMask segmented images were the CNN's two-channel inputs. In [39], a method for fingertip detection and real-time hand gesture recognition based on RGBD modality and the use of 3DCNN network was proposed. In [40], the best performance was reported using RGBD data and a histogram of gradient (HOG) with an SVM as a classifier.…”