Multi-view Mouth Renderization for Assisting Lip-reading

Mattos, Andréa Britto; Oliveira, Dário Augusto Borges

doi:10.1145/3192714.3192824

Cited by 8 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Problems such as dictating messages to smartphones in noisy environments [2,3], using visual silent passwords [4][5][6], transcribing silent films [7,8], synthesizing sound based on lip movements for speech-impaired people [9][10][11][12], and analyzing lip appearances to help hearing-impaired people [13] are among the application areas of automated lip reading systems.…”

Section: Introductionmentioning

confidence: 99%

Automated Word-level Lip Reading using Convolutional Neural Networks with New Turkish Dataset

Pervan-Akman¹,

Berkol²,

Erdem³

2023

Preprint

View full text Add to dashboard Cite

Automated lip reading is a research problem that has developed considerably in recent years. Lip reading is evaluated both visually and audibly in some cases. The lip reading model is a field of use for detecting specific words using images from security cameras, but it is not possible to use audio-visual databases in this situation. It is not possible to obtain the sound input of the pronounced word in all cases. We collected a new Turkish dataset with only the image in this study. The new dataset is produced using Youtube videos, which is an uncontrolled environment. For this reason, images have difficult parameters in terms of environmental factors such as light, angle, color, and personal characteristics of the face. Despite the different features on the human face such as mustache, beard, and make-up, the visual speech recognition problem was developed on 10 classes including single words and two-word phrases using Convolutional Neural Networks (CNN) without any intervention on the data. The proposed study using only-visual data obtained a model which is automated visual speech recognition with a deep learning approach. In addition, since this study uses only-visual data, the computational cost and resource usage is less than in multi-modal studies. It is also the first known study to address the lip reading problem with a deep learning algorithm using a new dataset belonging to the Ural-Altaic languages.

show abstract

Section: Introductionmentioning

confidence: 99%

Automated Word-level Lip Reading using Convolutional Neural Networks with New Turkish Dataset

Pervan-Akman¹,

Berkol²,

Erdem³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, due to the complexity of image processing and the difficulty of training classifiers, it is difficult for traditional lip‐reading systems to meet the requirements of real‐time applications. As a result of the advancements in lip‐reading systems, numerous applications are conceivable, for example, resolving multi‐talker simultaneous speech [10], developing augmented lip views to assist people with hearing impairments [11], dictating messages to smartphones in noisy environments [12], transcribing and re‐dubbing silent films [8], and discriminating between native and non‐native speakers [13].…”

Section: Introductionmentioning

confidence: 99%

Developing phoneme‐based lip‐reading sentences system for silent speech recognition

El-Bialy

Chen

Fenghour

et al. 2022

CAAI Trans on Intel Tech

View full text Add to dashboard Cite

Lip-reading is a process of interpreting speech by visually analysing lip movements. Recent research in this area has shifted from simple word recognition to lip-reading sentences in the wild. This paper attempts to use phonemes as a classification schema for lip-reading sentences to explore an alternative schema and to enhance system performance. Different classification schemas have been investigated, including characterbased and visemes-based schemas. The visual front-end model of the system consists of a Spatial-Temporal (3D) convolution followed by a 2D ResNet. Transformers utilise multi-headed attention for phoneme recognition models. For the language model, a Recurrent Neural Network is used. The performance of the proposed system has been testified with the BBC Lip Reading Sentences 2 (LRS2) benchmark dataset. Compared with the state-of-the-art approaches in lip-reading sentences, the proposed system has demonstrated an improved performance by a 10% lower word error rate on average under varying illumination ratios. K E Y W O R D S deep learning, deep neural networks, lip-reading, phoneme-based lip-reading, spatial-temporal convolution, transformers This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Artificial neural network technology for lips reading

Pyataeva

Dzyuba

2021

E3S Web Conf.

View full text Add to dashboard Cite

The paper presents the use of neural networks for the task of automated speech reading by lips articulation. Speech recognition is performed in two stages. First, a face search is performed and the lips area is selected in a separate frame of the video sequence using Haar features. Then the sequence of frames goes to the input of deep learning convolutional and recurrent neural networks for speech viseme recognition. Experimental studies were carried out using independently obtained videos with Russian-speaking speakers.

show abstract

Multi-view Mouth Renderization for Assisting Lip-reading

Cited by 8 publications

References 30 publications

Automated Word-level Lip Reading using Convolutional Neural Networks with New Turkish Dataset

Automated Word-level Lip Reading using Convolutional Neural Networks with New Turkish Dataset

Developing phoneme‐based lip‐reading sentences system for silent speech recognition

Artificial neural network technology for lips reading

Contact Info

Product

Resources

About