Enabling Two-Way Communication of Deaf Using Saudi Sign Language

Abstract

Disabled people are facing many difficulties communicating with others and involving in society. Modern societies have dedicated significant efforts to promote the integration of disabled individuals into their societies and services. Currently, smart healthcare systems are used to facilitate disabled people. The objective of this paper is to enable two-way communication of deaf individuals with the rest of society, thus enabling their migration from marginal elements of society to mainstream contributing elements. In the proposed system, we developed three modules; the sign recognition module (SRM) that recognizes the signs of a deaf individual, the speech recognition and synthesis module (SRSM) that processes the speech of a non-deaf individual and converts it to text, and an Avatar module (AM) to generate and perform the corresponding sign of the non-deaf speech, which were integrated into the sign translation companion system called Saudi deaf companion system (SDCS) to facilitate the communication from the deaf to the hearing and vice versa. This paper also contributes to the literature by utilizing our self-developed database, the largest Saudi Sign Language (SSL) database-the King Saud University Saudi-SSL (KSU-SSL). The proposed SDCS system performs 293 Saudi signs that are recommended by the Saudi Association for Hearing Impairment (SAHI) from 10 domains (healthcare, common, alphabets, verbs, pronouns and adverbs, numbers, days, kings, family, and regions).

Efhamni: A Deep Learning-Based Saudi Sign Language Recognition Application

“…Regarding the models used, most of the approaches considered a CNN variant, as they provided the best performance and are widely used in such applications. For instance, approaches that have considered utilizing CNN for ArSL/SSL are [3,[10][11][12], and for non-ArSL, approaches are [23,35]. Therefore, we have chosen to use CNN in our system.…”

“…The first category includes approaches that translate between the sign language and the text or audio in both directions. Only a few approaches fall into this category, such as the work of Faisal et al [12] and Shinde et al [17]. The second category includes approaches that generate sign languages from text or audio.…”

“…Faisal et al [12] developed a system called the Saudi Deaf Companion System (SDCS), which facilitates communication from the deaf to the hearing and vice versa. The system includes a module called the Sign Recognition Module (SRM), which is responsible for recognizing the signs made by deaf individuals and converting them into text.…”

“…To develop such automated models, deep semantic analysis is required, which is challenging due to the limited number and the small size of sign language datasets [5]. Few research studies, however, have targeted the ArSL due to its complexity compared to the non-Arabic SL and the data unavailability (e.g., [9]), and even fewer have targeted the Saudi Sign Language (SSL) (e.g., [3,[10][11][12][13][14][15]).…”