Exploring ROI size in deep learning based lipreading

Abstract

Automatic speechreading systems have increasingly exploited deep learning advances, resulting in dramatic gains over traditional methods. State-of-the-art systems typically employ convolutional neural networks (CNNs), operating on a video region-of-interest (ROI) that contains the speaker's mouth. However, little or no attention has been paid to the effects of ROI physical coverage and resolution on the resulting recognition performance within the deep learning framework. In this paper, we investigate such choices for a visual-only speech recognition system based on CNNs and long short-term memory models that we present in detail. Further, we employ a separate CNN to perform face detection and facial landmark localization, driving the ROI extraction process. We conduct experiments on a multi-speaker corpus of connected digits utterances, recorded in ideal visual conditions. Our results show that ROI design choices affect automatic speechreading performance significantly: the best visual-only word error rate (5.07%) corresponds to a ROI that contains a large part of the lower face, in addition to just the mouth, and at a relatively high resolution. Noticeably, the result represents a 27% relative error reduction compared to employing the entire lower face as the ROI.

A Survey of Research on Lipreading Technology

“…Nowadays, most lipreading research is to extract a fixed size of lip ROI as input, and the size of this size is still an open problem. Koumparoulis et al [207] proved in the experiment that the selection of ROI of different sizes of lips will have an impact on the final recognition results, but still cannot determine the optimal ROI size selection scheme. Many people talk.…”

Large-Scale Visual Speech Recognition

“…LipNet is the closest model to our neural network. Several similar architectures were subsequently introduced in the works of Thanda & Venkatesan (2017) who study audio-visual feature fusion, Koumparoulis et al (2017) who work on a small subset of 18 phonemes and 11 words to predict digit sequences, and Xu et al (2018) who presented a model cascading CTC with attention. were the first to use sequence-to-sequence models with attention to tackle audiovisual speech recognition with a real-world dataset.…”

MobiLipNet: Resource-Efficient Deep Learning Based Lipreading

“…Further, except for the pointwise-only network, all networks keep the conventional convolution in their first layer, i.e., without factorizing it to depthwise and pointwise ones. All models (including the baseline and ResNet) are trained on the same ROI size, as this affects both performance [45] and efficiency. In more detail, the following models are considered:…”