Development of Novel Lip-Reading Recognition Algorithm

Lin, Bor-Shing; Yao, Yu-Hsien; Liu, Ching-Feng; Lien, Chia-Yi; Lin, Bor‐Shyh

doi:10.1109/access.2017.2649838

Cited by 21 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We briefly considered the main challenges in the area (Katsaggelos et al, 2015), such as speaker dependency, pose variation and temporal information. We identified the most prospective approaches for visual features extraction, namely pixel-based and geometry-based methods (Lin et al, 2017). As round-up, we pointed out the most widely used visual speech modeling methods, which turned out to be support vector machines (SVM), hidden Markov models (HMM) or deep neural networks (DNN) based approaches (Lu et al, 2020).…”

Section: Backgrounds and Related Researchmentioning

confidence: 99%

A Novel Task-Oriented Approach Toward Automated Lip-Reading System Implementation

Ivanko

Ryumin

2021

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

Abstract. Visual information plays a key role in automatic speech recognition (ASR) when audio is corrupted by background noise, or even inaccessible. Speech recognition using visual information is called lip-reading. The initial idea of visual speech recognition comes from humans’ experience: we are able to recognize spoken words from the observation of a speaker's face without or with limited access to the sound part of the voice. Based on the conducted experimental evaluations as well as on analysis of the research field we propose a novel task-oriented approach towards practical lip-reading system implementation. Its main purpose is to be some kind of a roadmap for researchers who need to build a reliable visual speech recognition system for their task. In a rough approximation, we can divide the task of lip-reading into two parts, depending on the complexity of the problem. First, if we need to recognize isolated words, numbers or small phrases (e.g. Telephone numbers with a strict grammar or keywords). Or second, if we need to recognize continuous speech (phrases or sentences). All these stages disclosed in detail in this paper. Based on the proposed approach we implemented from scratch automatic visual speech recognition systems of three different architectures: GMM-CHMM, DNN-HMM and purely End-to-end. A description of the methodology, tools, step-by-step development and all necessary parameters are disclosed in detail in current paper. It is worth noting that for the Russian speech recognition, such systems were created for the first time.

show abstract

Section: Backgrounds and Related Researchmentioning

confidence: 99%

A Novel Task-Oriented Approach Toward Automated Lip-Reading System Implementation

Ivanko

Ryumin

2021

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…The SAVE dataset includes recorded videos of Arabic visemes using the NTST standard of 30 frames per seconds [30][31]. The Audio files are also included for validation of the output of the lip reading recognition system.…”

Section: A Viseme Feature Extractionmentioning

confidence: 99%

Novel Algorithm Utilizing Deep Learning for Enhanced Arabic Lip Reading Recognition

Khafaga¹,

Mahmoud²,

Alghamdi³

et al. 2021

IJACSA

View full text Add to dashboard Cite

Computerized lip reading is the science of translating visemes and oral lip reading into written text, where visemes are lip movement without sound. Video processing is applied for the recognition of those visemes. Previous research developed automated systems to for computerized lip reading recognition to be hearing-impaired aid. Many challenges face such an automation process, including insufficient training datasets. Also, speaker-dependency is one of the challenges that are faced. Real-time applications which respond within a specific time period are also widely required. Real-time human computer interaction are systems which require real time response. Response time for human computer interaction is measured by number of elapsed video frames. Video processing of lip reading necessitates real-time implementation. There are applications for viseme recognition, as an aid for deaf people, video games with human computer interaction, and surveillance systems. In this paper, a real-time viseme recognition system is introduced. In order to enhance existing methods to overcome these pitfalls, this paper proposed a computerized lip reading technique based on feature extraction. We utilized blocks arrangement techniques to reach a near-optimal appearance feature extraction technique. A Deep Neural Network is utilized to enhance recognition. The benchmark dataset SAVE, for Arabic visemes, is employed in this research, and high viseme recognition accuracies are achieved. The described computerized lip reading recognition technique is advantageous for the hearing impaired and for other speakers in noisy environments.

show abstract

“…In paper [4], the lip contours and the English vowels were recognized when spoken by taking several lip features such as the lip contours and the ratio of width/height. This project was also able to identify the Region of Interest near the mouth on its own.…”

Section: Literature Reviewmentioning

confidence: 99%

LipVision: A Deep Learning Approach

Khetarpal¹,

Moradian²,

Sadar³

et al. 2017

IJCA

View full text Add to dashboard Cite

LipReading is the task of interpreting what an individual is saying by analysing his/her mouth patterns while the individual is talking. The paper is conducting a survey on the previously done work on LipReading. It will be discussing the different classifiers used, their efficiency and the end accuracy obtained. LipReading can be used in a myriad of fields such as medical, communication and gaming. The proposed system will use the GRID corpus dataset in which the videos are recorded from 33 speakers. OpenCV and dlib will be used for face and mouth detection. Then the mouth ROI will be used with the iBug tool to annotate facial landmarks. The architecture consists of Convolutional Neural Networks which will be created and trained in Tensorflow (Open Source Software Library), which are then passed through Connectionist Temporal Classification. It will then be using saliency visualisation technique to interpret and match the learned behaviour and generate text.

show abstract

Development of Novel Lip-Reading Recognition Algorithm

Cited by 21 publications

References 25 publications

A Novel Task-Oriented Approach Toward Automated Lip-Reading System Implementation

A Novel Task-Oriented Approach Toward Automated Lip-Reading System Implementation

Novel Algorithm Utilizing Deep Learning for Enhanced Arabic Lip Reading Recognition

LipVision: A Deep Learning Approach

Contact Info

Product

Resources

About