Deep learning based assistive technology on audio visual speech recognition for hearing impaired

Kumar, L. Ashok; Renuka, D. Karthika; Rose, S. Lovelyn; Priya, M.C. Shunmuga; Wartana, I Made

doi:10.1016/j.ijcce.2022.01.003

Cited by 50 publications

(20 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The paper has presented current state of the audio-visual speech recognition area as well as potential research directions. Kumar et al (2022) proposed a deep learning technique-based audiovisual speech recognition system for hearing impaired people. Hearing challenged students confront several problems, including a lack of skilled sign language facilitators and the expensive cost of assistive technology.…”

Section: Literature Reviewsmentioning

confidence: 99%

RETRACTED ARTICLE: Audio-Visual Automatic Speech Recognition Towards Education for Disabilities

Debnath

Roy

Namasudra

et al. 2022

J Autism Dev Disord

View full text Add to dashboard Cite

Education is a fundamental right that enriches everyone's life. However, physically challenged people often debar from the general and advanced education system. Audio-Visual Automatic Speech Recognition (AV-ASR) based system is useful to improve the education of physically challenged people by providing hands-free computing. They can communicate to the learning system through AV-ASR. However, it is challenging to trace the lip correctly for visual modality. Thus, this paper addresses the appearance-based visual feature along with the co-occurrence statistical measure for visual speech recognition. Local Binary Pattern-Three Orthogonal Planes (LBP-TOP) and Grey-Level Co-occurrence Matrix (GLCM) is proposed for visual speech information. The experimental results show that the proposed system achieves 76.60 % accuracy for visual speech and 96.00 % accuracy for audio speech recognition.

show abstract

Section: Literature Reviewsmentioning

confidence: 99%

RETRACTED ARTICLE: Audio-Visual Automatic Speech Recognition Towards Education for Disabilities

Debnath

Roy

Namasudra

et al. 2022

J Autism Dev Disord

View full text Add to dashboard Cite

show abstract

“…Analyzing a student's performance using NB classifier is one of the classification methods used to recognize hidden relations between subjects in Sijil Pelajaran Malaysia. The algorithm can be implemented for performance classification during the early stage of 2nd semester achieving an accuracy of 74% [12]. Works implemented in [13] involved building a recurrent neural network (RNN) for forecasting students' final grades from log information in education systems.…”

Section: A Literature Reviewmentioning

confidence: 99%

HFIPO-DPNN: A Framework for Predicting the Dropout of Physically Impaired Student from Education

Marina.¹,

Senthilrajan²

2023

IJIET

View full text Add to dashboard Cite

Education plays a significant role in individuals’ development and the economic growth of developing countries like India. Dropout of students from their studies is the major concern for any order of education. Some models for predicting the dropout of students are developed with several factors. Many of them lacked consistency as they backed their studies with the academic performance of the students. Especially, for those students who suffered from physical impairment, the dropout depends on several external factors. Hence, this work proposes a novel HFIPO-DPNN to predict the student dropout rooted in the previous semester’s marks. The proposed model enclosed the hybrid firefly and improved particle swarm algorithm to optimize the feature selection that influences the dropout of hearing-impaired students. The optimized feature data are used to predict the dropout with the novel DPNN. The optimized data was split and used for training the DPNN. The testing data is used to evaluate the performance of the proposed framework. The attributes used for predicting the student dropout are Family Size, Subject, Medium of Instruction, and so on. The data must be collected from 250 physically impaired children belonging to ITI institute, Bangalore. The outcome of the proposed framework is evaluated on several metrics. The accuracy of the proposed model is about 99.02%. The HFIPO-DPNN framework can be enhanced for predicting the dropout for students with other disabilities. The optimization showed that factors influencing education other than familial factors are to be considered in the prediction of dropout.

show abstract

“…CNN is now taking the place of traditional MFCC, DCT, and AAM in aspects of audio and visual feature extractions. Also, HMM and MSHMM are currently being substituted by Long Short-Time Memory networks (LSTM) or Bidirectional Long-Short-Term Memory networks (Bi-LSTM) in aspects of time sequence modeling [83,84,85]. As the audio information is not completely available for the hearing-impaired, the use of a relevant acoustic feature extraction method is very significant for AVSR.…”

Section: Audio-visual Speech Recognitionmentioning

confidence: 99%

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Bhaskar

Madathil

2022

IJEEI

View full text Add to dashboard Cite

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures-facial, lip-hand position, etc., are mainly used as sensory modalities to develop hard-of-hearing multimodal systems. This paper encapsulates an overview of multimodal systems available within the literature for hard-ofhearing studies. This paper also discusses some of the studies related to hardof-hearing acoustic analysis. It is observed that very few algorithms have been developed for hard-of-hearing AVSR than normal hearing. Thus, the study of audio-visual-based speech recognition systems for hard-of-hearing is highly demanded by people trying to communicate with natively speaking languages. This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers in the development of AVSR systems.

show abstract

Deep learning based assistive technology on audio visual speech recognition for hearing impaired

Cited by 50 publications

References 11 publications

RETRACTED ARTICLE: Audio-Visual Automatic Speech Recognition Towards Education for Disabilities

RETRACTED ARTICLE: Audio-Visual Automatic Speech Recognition Towards Education for Disabilities

HFIPO-DPNN: A Framework for Predicting the Dropout of Physically Impaired Student from Education

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Contact Info

Product

Resources

About