Multi-pose lipreading and audio-visual speech recognition

Estellers, Virginia; Thiran, Jean-Philippe

doi:10.1186/1687-6180-2012-51

Cited by 21 publications

(3 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different views of the speaker were used for lip reading using a pose normalization block in a standard system. The effects of pose normalization on the audiovisual integration strategy are analyzed by AV-ASR [16].…”

Section: Literature Surveymentioning

confidence: 99%

Visual Speech Recognition using VGG16 Convolutional Neural Network

Kulkarni¹,

Murthy²

2021

Preprint

View full text Add to dashboard Cite

Communication is all about expressing one’s thoughts to another person through speech and facial expressions. But for people with hearing impairment, it is difficult to communicate without any assistance. In most of these cases Visual speech recognition (VSR) systems simplify the tasks by using Machine Learning algorithms and assisting them to understand speech and socialize without depending on the auditory perception. Thus, one can visualize VSR system as a lifeline for people with hearing impairment which helps them in providing a way to understand the words that are being tried to convey to them through speech. In this work we used VGG16 convolutional neural network architecture for Kannada and English datasets. We used custom dataset for the research work and got the accuracy of 90.10% for English database and 91.90% for Kannada database.

show abstract

Section: Literature Surveymentioning

confidence: 99%

Visual Speech Recognition using VGG16 Convolutional Neural Network

Kulkarni¹,

Murthy²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Lucey et al [12] apply a linear mapping to transform profile view features to frontal view features. This approach has been extended to map other views like 30°, 45°and 60°to the frontal view [13] or to the 30°v iew [14]. However, the performance is degraded as the number of features to be generated by the linear mapping increases [12].…”

Section: Introductionmentioning

confidence: 99%

Towards Pose-Invariant Lip-Reading

Cheng

Tzimiropoulos

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Lip-reading models have been significantly improved recently thanks to powerful deep learning architectures. However, most works focused on frontal or near frontal views of the mouth. As a consequence, lip-reading performance seriously deteriorates in non-frontal mouth views. In this work, we present a framework for training pose-invariant lip-reading models on synthetic data instead of collecting and annotating non-frontal data which is costly and tedious. The proposed model significantly outperforms previous approaches on nonfrontal views while retaining the superior performance on frontal and near frontal mouth views. Specifically, we propose to use a 3D Morphable Model (3DMM) to augment LRW, an existing large-scale but mostly frontal dataset, by generating synthetic facial data in arbitrary poses. The newly derived dataset, is used to train a state-of-the-art neural network for lip-reading. We conducted a cross-database experiment for isolated word recognition on the LRS2 dataset, and reported an absolute improvement of 2.55%. The benefit of the proposed approach becomes clearer in extreme poses where an absolute improvement of up to 20.64% over the baseline is achieved.

show abstract

“…However, there is a lot of interest to address, for example, the problem of head pose, which is a large hindrance in the application to real-world scenarios. Various works have already addressed this problem by taking different view angles into account [18,8]. To this end, several databases have been recorded simultaneously with cameras at different angles [17,11].…”

Section: Introductionmentioning

confidence: 99%

Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System

Zimmermann

Ghazi

Ekenel

et al. 2017

Computer Vision – ACCV 2016 Workshops

Self Cite

View full text Add to dashboard Cite

Abstract. Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches -extracted from the video data -to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.The final publication is available at Springer via http://dx

show abstract

Multi-pose lipreading and audio-visual speech recognition

Cited by 21 publications

References 44 publications

Visual Speech Recognition using VGG16 Convolutional Neural Network

Visual Speech Recognition using VGG16 Convolutional Neural Network

Towards Pose-Invariant Lip-Reading

Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System

Contact Info

Product

Resources

About