A Novel Technique for Speech Recognition and Visualization Based Mobile Application to Support Two-Way Communication between Deaf-Mute and Normal Peoples

Abstract

Mobile technology is very fast growing and incredible, yet there are not much technology development and improvement for Deafmute peoples. Existing mobile applications use sign language as the only option for communication with them. Before our article, no such application (app) that uses the disrupted speech of Deaf-mutes for the purpose of social connectivity exists in the mobile market. The proposed application, named as vocalizer to mute (V2M), uses automatic speech recognition (ASR) methodology to recognize the speech of Deaf-mute and convert it into a recognizable form of speech for a normal person. In this work mel frequency cepstral coefficients (MFCC) based features are extracted for each training and testing sample of Deaf-mute speech. The hidden Markov model toolkit (HTK) is used for the process of speech recognition. The application is also integrated with a 3D avatar for providing visualization support. The avatar is responsible for performing the sign language on behalf of a person with no awareness of Deaf-mute culture. The prototype application was piloted in social welfare institute for Deaf-mute children. Participants were 15 children aged between 7 and 13 years. The experimental results show the accuracy of the proposed application as 97.9%. The quantitative and qualitative analysis of results also revealed that face-to-face socialization of Deaf-mute is improved by the intervention of mobile technology. The participants also suggested that the proposed mobile application can act as a voice for them and they can socialize with friends and family by using this app.

Speech to Text in Indonesian Personal Assistant

“…The used method is Mel Frequency Cepstral Coefficient (MFCC) for feature extraction of training, Hidden Markov Model (HMM) Toolkit for recognition process and also 3D avatar to create sign language visualization. The result obtained is 97.9% for 15 participants from deaf-mute children social foundation (Yousaf et al, 2018).…”

“…Based on Mittal and Navdeep (2016) some of these challenges include processing power, memory usage, an accuracy of speech signal recognition, use of time and energy consumption (Mittal and Navdeep, 2016). Furthermore, similar technology was developed by Yousaf et al (2018) that is called Vocalizer to Mute (V2M). The novelty presented in this research is the integration between speech recognition and 3D avatar animation visualization to support deafmute communication.…”

Understanding and Designing for Deaf or Hard of Hearing Drivers on Uber

“…According to a study done by Bragg et al [9], Deaf people would rather receive many notifications than miss a notification. While all these studies and product examples focus on sound detection and notification, little research has been done related to the context of driving, and in particular, the use of a ridesharing platform through which notifications are frequently received [29] prototyped and evaluated a mobile phone application that utilizes speech-to-text and text-tosign language to visualize the sign language using an avatar and convert the sign language to text. This enables DHH individuals and hearing people to communicate.…”

“…The researchers saw the need for a new strategy and suggested Automatic Speech Recognition (ASR) as an option. Yousaf et al [29] also investigated DHH individuals' perceptions of captioning imperfect ASR in oneon-one meetings. Elliot et al [12] conducted a follow-up study in which they investigated a messaging application that incorporated ASR and assessed its effectiveness for communication between a DHH individual (typing) and a hearing individual (speech and ASR).…”

Spatio-Temporal Residual Networks for Slide Transition Detection in Lecture Videos

“…Lin et al [16] combined speech text transcript, audio and video information to design an automated segmentation approach. Speech Recognition [17] can be also applied to segment the audio signal for lecture video summary. Qazi et al [18] introduced a hybrid technique for speech segregation and calssification using the deep belief network (DBN) model.…”