Automatic Lip-Reading of Hearing Impaired People

Ivanko, Denis; Ryumin, Dmitry; Karpov, Alexey

doi:10.5194/isprs-archives-xlii-2-w12-97-2019

Cited by 13 publications

(7 citation statements)

References 16 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Convolution Neural Network (CNN) was used to train audios and images. As the first step, the conversion of the Sign Language into audio using CNN was determined [11,12,13]. Using CNN can achieve accurate results.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A Multi-Modular Approach for Sign Language and Speech Recognition for Deaf-Mute People

2021

View full text Add to dashboard Cite

Deaf and Mute people cannot communicate efficiently to express their feelings to ordinary people. The common method these people use for communication is the sign language. But these sign languages are not very familiar to ordinary people. Therefore, effective communication between deaf and mute people and ordinary people is seriously affected. This paper presents the development of an Android mobile application to translate sign language into speech-language for ordinary people, and speech into text for deaf and mute people using Convolution Neural Network (CNN). The study focuses on vision-based Sign Language Recognition (SLR) and Automatic Speech Recognition (ASR) mobile application. The main challenging tasks were audio classification and image classification. Therefore, CNN was used to train audio clips and images. Mel-frequency Cepstral Coefficient (MFCC) approach was used for ASR. The mobile application was developed by Python programming and Android Studio. After developing the application, testing was done for letters A and C, and these letters were identified with 95% accuracy.

show abstract

Section: Methodsmentioning

confidence: 99%

“…It is a challenging task since the human voice signals are inconsistent due to change in speaker qualities, different speaking styles, uncertain background noises, etc. [11].…”

Section: Introductionmentioning

confidence: 99%

A Multi-Modular Approach for Sign Language and Speech Recognition for Deaf-Mute People

2021

View full text Add to dashboard Cite

show abstract

“…При этом разработчики сталкиваются с рядом сложностей, среди которых наклон и резкий поворот головы или перекрытие лица руками. Ввиду подобных сложностей, дополнительный анализ мимики может сводиться к оценке движений отдельных частей лица, например, как в работах [121,122] анализ движений рта.…”

Section: рис 3 основные методы автоматического распознавания устной р...unclassified

Modern automatic recognition technologies for visual communication tools

Yachnaya¹,

Lutsiv²,

Malashin³

2023

Computer Optics

View full text Add to dashboard Cite

Communication refers to a wide range of different behaviors and activities aimed at handing over information. The communication process includes verbal, paraverbal and non-verbal components, conveying the informational part of a message and its emotional part respectively. A complex analysis of all communication components makes it possible to evaluate not only the content, but also the situational context of what is being said, as well as to identify additional factors inherent in the mental and somatic state of the speaker. There are several methods of conveying a verbal message, among which are oral and gestural speech (such as the sign language and fingerspelling). Various forms of communication can be contained in multiple data transmission channels, such as audio or video channels. The review is concerned with video data analysis systems, as the audio channel is incapable of non-verbal components transmission that contribute supplemental details. The article analyzes databases of static and dynamic images and systems, developed to recognize the verbal component conveyed by oral and gestural speech, as well as systems that evaluate paraverbal and non-verbal components of communication. Challenges of designing such databases and systems are specified. Prospective directions in complex analysis of all communication components and its combinations for the most complete evaluation of messages are defined.

show abstract

“…In addition, one in three people over the age of 65 experience hearing loss, and it is estimated that more than 2 billion people will be deaf or hard of hearing by 2050. Therefore, intelligent technologies (systems) of effective automatic machine sign language recognition (SLR) are needed to organize a natural HCI [ 7 ].…”

Section: Introductionmentioning

confidence: 99%

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Ryumin

Ivanko

Ryumina

2023

Sensors

View full text Add to dashboard Cite

Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human–computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora—LRW and AUTSL—and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.

show abstract

Automatic Lip-Reading of Hearing Impaired People

Cited by 13 publications

References 16 publications

A Multi-Modular Approach for Sign Language and Speech Recognition for Deaf-Mute People

A Multi-Modular Approach for Sign Language and Speech Recognition for Deaf-Mute People

Modern automatic recognition technologies for visual communication tools

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Contact Info

Product

Resources

About