Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks

Abstract

Dynamic hand gesture recognition is a challenging task of Human-Computer Interaction (HCI) and Computer Vision. The potential application areas of gesture recognition include sign language translation, video gaming, video surveillance, robotics, and gesture-controlled home appliances. In the proposed research, gesture recognition is applied to recognize sign language words from real-time videos. Classifying the actions from video sequences requires both spatial and temporal features. The proposed system handles the former by the Convolutional Neural Network (CNN), which is the core of several computer vision solutions and the latter by the Recurrent Neural Network (RNN), which is more efficient in handling the sequences of movements. Thus, the real-time Indian sign language (ISL) recognition system is developed using the hybrid CNN-RNN architecture. The system is trained with the proposed CasTalk-ISL dataset. The ultimate purpose of the presented research is to deploy a real-time sign language translator to break the hurdles present in the communication between hearing-impaired people and normal people. The developed system achieves 95.99% top-1 accuracy and 99.46% top-3 accuracy on the test dataset. The obtained results outperform the existing approaches using various deep models on different datasets.

Spatiotemporal Modeling for Dynamic Gesture Recognition in Video Streams

“…CSL-Chinese Sign Language, ASL-American Sign LanguageAs per literature 3D CNN & BiLSTM methods[21],[18],[19] proposed on Chinese Sign language are giving promising results out of which BiLSTM is giving better results. Hence, we used pretrained networks with BiLSTM on custom made ISL data set consisting of words and phrases and could achieve better recognition accuracy[Table:3].…”

“…Hence, we used pretrained networks with BiLSTM on custom made ISL data set consisting of words and phrases and could achieve better recognition accuracy[Table:3]. Inception v3 + LSTM on CasTalk ISL[18] also gave similar results but the size of database is less and also it consists of only words. Another work on ASL Database[20] gave better recognition accuracy on DHG 14 but the number of gestures is less.…”

A Communication Translator Interface for Sign Language Interpretation

“…Both SVM and CNN classifiers exhibited high accuracy rates on the testing set of images, with SVM achieving 99.14% accuracy on test data and an overall accuracy of 99% for alphabets and digits. A team proposed the development of a real-time sign language recognition system utilizing a hybrid CNN-RNN architecture to identify sign language words from real-time videos [2]. The system achieved remarkable results with a top-1 accuracy of 95.99% and a top-3 accuracy of 99.46% on the test dataset.…”