A Multi-Modular Approach for Sign Language and Speech Recognition for Deaf-Mute People

Abstract

Deaf and Mute people cannot communicate efficiently to express their feelings to ordinary people. The common method these people use for communication is the sign language. But these sign languages are not very familiar to ordinary people. Therefore, effective communication between deaf and mute people and ordinary people is seriously affected. This paper presents the development of an Android mobile application to translate sign language into speech-language for ordinary people, and speech into text for deaf and mute people using Convolution Neural Network (CNN). The study focuses on vision-based Sign Language Recognition (SLR) and Automatic Speech Recognition (ASR) mobile application. The main challenging tasks were audio classification and image classification. Therefore, CNN was used to train audio clips and images. Mel-frequency Cepstral Coefficient (MFCC) approach was used for ASR. The mobile application was developed by Python programming and Android Studio. After developing the application, testing was done for letters A and C, and these letters were identified with 95% accuracy.

Advancements in Vision-Based Sign Language Recognition: A Comprehensive Review

“…Similarly, research [7] used a webcam, and they were able to capture 30 fps with a 0.7-megapixel resolution, and [8] used a 15-fps frame rate. Since some researchers have implemented mobile applications, they have used the phone camera to capture data [9]. In research, [10] Wimalaratne and Fernando used a Kinect XBOX 360 device to capture some additional information as the depth of an image.…”

“…Since the input data of a system may have varying heights, hand lengths, and other differences, it's essential to normalize the data properly to ensure that the final output remains unaffected by these factors. The input images were converted into a fixed size in [9], [16], [17], [11] and [18]. In some research articles, [10], [19] images were normalized at the pre-processing stage.…”

“…In [9], Dahanayaka et al focused on achieving real-time object detection. They accomplished this by developing a Single-Shot Detector (SSD) VGG16 model designed for swift object detection.…”

“…Similarly, [6], [16], [7], and [9] also used CNN-based approaches for classification where image-based inputs of the gestures are used. In [16], De Silva et al have only used simple static word signs, where each gesture is recognized individually and generated results for each word separately.…”

“…In [16], De Silva et al have only used simple static word signs, where each gesture is recognized individually and generated results for each word separately. [9] and [6] also followed a similar approach to De Silva et al for letters. In [7], they have recognized sentences using finger spelling signing.…”