Hand Gesture Recognition for Sign Language Using 3DCNN

Abstract

Recently, automatic hand gesture recognition has gained increasing importance for two principal reasons: the growth of the deaf and hearing-impaired population, and the development of visionbased applications and touchless control on ubiquitous devices. As hand gesture recognition is at the core of sign language analysis a robust hand gesture recognition system should consider both spatial and temporal features. Unfortunately, finding discriminative spatiotemporal descriptors for a hand gesture sequence is not a trivial task. In this study, we proposed an efficient deep convolutional neural networks approach for hand gesture recognition. The proposed approach employed transfer learning to beat the scarcity of a large labeled hand gesture dataset. We evaluated it using three gesture datasets from color videos: 40, 23, and 10 classes were used from these datasets. The approach obtained recognition rates of 98.12%, 100%, and 76.67% on the three datasets, respectively for the signer-dependent mode. For the signer-independent mode, it obtained recognition rates of 84.38%, 34.9%, and 70% on the three datasets, respectively. INDEX TERMS 3DCNN, computer vision, deep learning, hand gesture recognition, sign language recognition, transfer learning.

Deep Learning-Based Approach for Sign Language Gesture Recognition With Efficient Hand Gesture Representation

“…This consideration led to excellent improvement in system performance. Compared to the results achieved by the base C3D architecture in the first experiment and those achieved by the temporally enhanced system in [20], this system achieved the best recognition rate with both MLP and autoencoders in all the scenarios.…”

“…The good performance achieved by the systems in [20] and [37] can be attributed to the efficient way of utilizing the temporal features of the hand gesture. In this regard, the system in [20] utilized 3DCNN to model three temporal segments, from the beginning, the middle, and the end of the input video and then aggregated the segments' features to achieve a robust temporal representation.…”

“…Our experiments were conducted on the King Saud University Saudi Sign Language (KSU-SSL) dataset reported in [20]. The dataset contains isolated words and phrases from common expressions in the SSL dictionary.…”

“…Two different techniques, i.e., MLP and autoencoder, are investigated to fuse the two feature vectors before feeding them to the classifier. In contrast to the system proposed in [20], we avoid the use of LSTM with this system because the two streams are not temporal segments of the gesture. Then, we perform end-to-end training for the fusion architecture with the classifier.…”

“…In other words, most of the frame area contains non-relevant features that increase the misclassification ratio. In another work, we addressed the first mentioned drawback of modeling the long-term temporal dependence [20] by using independent instances of 3DCNN to model the local spatiotemporal features of different temporal segments. We also explored different techniques to globalize the local representations.…”