MobiLipNet: Resource-Efficient Deep Learning Based Lipreading

Koumparoulis, Alexandros; Potamianos, Gerasimos

doi:10.21437/interspeech.2019-2618

Cited by 6 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Feng et al [127] improved this architecture by integrating the Squeeze-and-Extract [128] module. Besides VGG and ResNet, researchers have also adopted other representative 2D CNN architectures, including DenseNet [58], ShuffleNet [52], MobileNet [129], etc.…”

Section: Visual Frontend Networkmentioning

confidence: 99%

Deep Learning for Visual Speech Analysis: A Survey

Sheng¹,

Kuang²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper aims to present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.

show abstract

Section: Visual Frontend Networkmentioning

confidence: 99%

Deep Learning for Visual Speech Analysis: A Survey

Sheng¹,

Kuang²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This model consists of spatiotemporal convolutions and recurrent operations, and that is trained by a connectionist temporal classification loss [18]. MobiLipNet [19] has been proposed to achieve computationally efficient lip reading, and that uses the depthwise convolution and the pointwise convolution. There are some prior works based on a generative adversarial network (GAN) [20] for lip reading.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised domain adaptation for lip reading based on cross-modal knowledge distillation

Takashima

Tsunoda

et al. 2021

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

We present an unsupervised domain adaptation (UDA) method for a lip-reading model that is an image-based speech recognition model. Most of conventional UDA methods cannot be applied when the adaptation data consists of an unknown class, such as out-of-vocabulary words. In this paper, we propose a cross-modal knowledge distillation (KD)-based domain adaptation method, where we use the intermediate layer output in the audio-based speech recognition model as a teacher for the unlabeled adaptation data. Because the audio signal contains more information for recognizing speech than lip images, the knowledge of the audio-based model can be used as a powerful teacher in cases where the unlabeled adaptation data consists of audio-visual parallel data. In addition, because the proposed intermediate-layer-based KD can express the teacher as the sub-class (sub-word)-level representation, this method allows us to use the data of unknown classes for the adaptation. Through experiments on an image-based word recognition task, we demonstrate that the proposed approach can not only improve the UDA performance but can also use the unknown-class adaptation data.

show abstract

“…and battery consumption are also important factors. As a consequence, few works have also focused on the computational complexity of visual speech recognition [11,12], but such models still trail massively behind full-fledged ones in terms of accuracy.…”

Section: Introductionmentioning

confidence: 99%

“…Such models consist of fully connected [4,5,6,7,8] or convolutional layers [9,10,11,12] which extract features from the mouth region of interest, followed by recurrent layers or attention [12,13] / self-attention architectures [11]. Few works have also focused on the computational complexity of visual speech recognition [14,15], and in any case efficient methods have trailed massively behind full-fledged ones in terms of accuracy.…”

Section: Introductionmentioning

confidence: 99%

Towards Practical Lipreading with Distilled and Efficient Models

Martínez

Petridis

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Lipreading has witnessed a lot of progress due to the resurgence of neural networks. Recent works have placed emphasis on aspects such as improving performance by finding the optimal architecture or improving generalization. However, there is still a significant gap between the current methodologies and the requirements for an effective deployment of lipreading in practical scenarios. In this work, we propose a series of innovations that significantly bridge that gap: first, we raise the state-of-the-art performance by a wide margin on LRW and LRW-1000 to 88.5 % and 46.6 %, respectively using self-distillation. Secondly, we propose a series of architectural changes, including a novel Depthwise Separable Temporal Convolutional Network (DS-TCN) head, that slashes the computational cost to a fraction of the (already quite efficient) original model. Thirdly, we show that knowledge distillation is a very effective tool for recovering performance of the lightweight models. This results in a range of models with different accuracy-efficiency trade-offs. However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8.2× and 3.9× in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.

show abstract

MobiLipNet: Resource-Efficient Deep Learning Based Lipreading

Cited by 6 publications

References 27 publications

Deep Learning for Visual Speech Analysis: A Survey

Deep Learning for Visual Speech Analysis: A Survey

Unsupervised domain adaptation for lip reading based on cross-modal knowledge distillation

Towards Practical Lipreading with Distilled and Efficient Models

Contact Info

Product

Resources

About