Arabic Sign Language Recognition using Faster R-CNN

Abstract

Deafness does not restrict its negative effect on the person's hearing, but rather on all aspect of their daily life. Moreover, hearing people aggravated the issue through their reluctance to learn sign language. This resulted in a constant need for human translators to assist deaf person which represents a real obstacle for their social life. Therefore, automatic sign language translation emerged as an urgent need for the community. The availability and the widespread use of mobile phones equipped with digital cameras promoted the design of image-based Arabic Sign Language (ArSL) recognition systems. In this work, we introduce a new ArSL recognition system that is able to localize and recognize the alphabet of the Arabic sign language using a Faster Region-based Convolutional Neural Network (R-CNN). Specifically, faster R-CNN is designed to extract and map the image features, and learn the position of the hand in a given image. Additionally, the proposed approach alleviates both challenges; the choice of the relevant features used to encode the sign visual descriptors, and the segmentation task intended to determine the hand region. For the implementation and the assessment of the proposed Faster R-CNN based sign recognition system, we exploited VGG-16 and ResNet-18 models, and we collected a real ArSL image dataset. The proposed approach yielded 93% accuracy and confirmed the robustness of the proposed model against drastic background variations in the captured scenes.

Deep Learning Technology to Recognize American Sign Language Alphabet

“…The proposed system achieved an accuracy of 90.02% by training 80% of dataset images. The research introduced in [ 31 ] aims to translate the hand gestures of two-dimensional images into text using a faster region-based convolutional neural network (R-CNN). Their system mapped the position of the hand gestures and recognized the letters.…”

A Deep Learning-Based Approach for Hand Sign Recognition Using CNN Architecture

“…Convolutional Neural Networks (CNNs), a type of deep learning neural network, are frequently employed for image identification and classification applications. CNNs are designed to automatically detect features in input images through a process called convolution, which extracts specific features such as edges, corners, and textures [7]. In sum, CNN architecture is chosen for Hand Sign Recognition due to its innate ability to capture spatial features and patterns within images, which aligns perfectly with the distinctive visual characteristics of hand signs, leading to superior recognition accuracy.…”

Azerbaijani sign language recognition using machine learning approach

“…Salma Hayani et al [5] presented a paper based on the LeNet-5 CNN architecture to recognize numbers and letters of Arab sign language (ArSL) automatically. Other ArSL recognition-related works are provided by Rahaf AbdulAziz Alawwad et al [6] and Ouiem Bchir [7]. Authors of [6] introduced ArSL detection system with the ability to localize and recognize the alphabet of the ArSL using a Faster R-CNN.…”

“…Other ArSL recognition-related works are provided by Rahaf AbdulAziz Alawwad et al [6] and Ouiem Bchir [7]. Authors of [6] introduced ArSL detection system with the ability to localize and recognize the alphabet of the ArSL using a Faster R-CNN. To identify ground truth data or for localization of the object authors used bounding box type.…”