Transforming Sign Language into Text and Speech through Deep Learning Technologies

Abstract

Objective:The goal of the proposed work is to leverage deep learning technologies to create an efficient and accurate system for transforming sign language into text and speech. People deliver their ideas, feelings, and experiences to others around them via their interactions with each other. The hand gesture plays a significant role since it reflects the user's thoughts more rapidly than other motions (head, face, eye, and body). For deaf-mute people with disabilities, this is still not the case. Sign language facilitates communication among deaf-mute individuals. An individual who is deafmute can communicate without the use of acoustic noises by using sign language. Methods: Convolutional neural networks (CNNs) are generally used to recognize and extract characteristics from sign language motions. These neural networks are employed to recognize and extract critical features from sign language gestures. These features are processed by natural language processing models for textual translation. Finally, neural text-to-speech (TTS) technology is used to translate the textual translations into synthesized speech, thereby bridging the communication gap for the Deaf community. To establish an inclusive and accessible communication system, this technique combines computer vision, natural language processing, and speech synthesis. Findings: The datasets used in this technique include hand gesture images, which contain different hand poses and expressions. It is used to train and assess the model. The experiment findings show an accuracy of 97.6% with a precision of 94.1%, a recall of 96.8%, and an F1-score of 95.9%. Novelty: This approach displays a cogent translation from text to speech and achieves an outstanding translation accuracy of 97.6% from sign language to text, producing a natural and understandable output.