Web Based Recognition and Translation of American Sign Language with CNN and RNN

Abstract

Individuals with hearing hindrance utilize gesture based communication to exchange their thoughts. Generally hand movements are used by them to communicate among themselves. But there are certain limitations when they communicate with other people who cannot understand these hand movements. There is a need to have a mechanism that can act as a translator between these people to communicate. It would be easier for these people to interact if there exists direct infrastructure that is able to convert signs to text and voice messages. As of late, numerous such frameworks for gesture based communication acknowledgment have been developed. But most of them are made either for static gesture recognition or dynamic gesture recognition. As sentences are generated using combinations of static and dynamic gestures, it would be simpler for hearing debilitated individuals if such computerized frameworks can detect both the static and dynamic motions together. We have proposed a design and architecture of American Sign Language (ASL) recognition with convolutional neural networks (CNN). This paper utilizes a pretrained VGG-16 architecture for static gesture recognition and for dynamic gesture recognition, spatiotemporal features were learnt with the complex architecture, called deep learning. It contains a bidirectional convolutional Long Short Term Memory network (ConvLSTM) and 3D convolutional neural network (3DCNN) and this architecture is responsible to extract 2D spatio temporal features.

Automatic Leukocytes Classification using Deep Convolutional Neural Network

“…A twodimensional convolution with a kernel size of 3x3 is applied to the input RGB image of 224x224 pixels during the training phase. The layers that make up the VGG16 network have the following specifications (Bendarkar et al, 2021) (Maysanjaya, 2020)…”

A Systematic Literature Review on Vision-Based Hand Gesture for Sign Language Translation

“…Most alphabet and number recognition are in static action. Like Al-Amin (2017); Bendarkar et al (2021) studies in the American Sign Language recognition system, all the alphabets are in static action, except j and z alphabet, which are in dynamic action.…”

Cutting-edge communication and learning assistive technologies for disabled children: An artificial intelligence perspective

“…Non-signers, i.e., people who are not familiar with sign language can communicate with deaf people who speak using the translators of sign language into text or speech (Truong et al, 2016 ). These translators predominantly use ML algorithms to find the correct sign, like convolutional and recurrent neural networks (Bendarkar et al, 2021 ) or deep learning (Bantupalli and Xie, 2018 ). A very promising human-machine interface (HMI) device are communication gloves, which have sensors that interpret the motions of sign languages into natural language combining virtual and augmented reality with AAC (Ozioko and Dahiya, 2022 ).…”