Flow-Based Unconstrained Lip to Speech Generation

Abstract

Unconstrained lip-to-speech aims to generate corresponding speeches based on silent facial videos with no restriction to head pose or vocabulary. It is desirable to generate intelligible and natural speech with a fast speed in unconstrained settings. Currently, to handle the more complicated scenarios, most existing methods adopt the autoregressive architecture, which is optimized with the MSE loss. Although these methods have achieved promising performance, they are prone to bring issues including high inference latency and mel-spectrogram over-smoothness. To tackle these problems, we propose a novel flow-based non-autoregressive lip-to-speech model (GlowLTS) to break autoregressive constraints and achieve faster inference. Concretely, we adopt a flow-based decoder which is optimized by maximizing the likelihood of the training data and is capable of more natural and fast speech generation. Moreover, we devise a condition module to improve the intelligibility of generated speech. We demonstrate the superiority of our proposed method through objective and subjective evaluation on Lip2Wav-Chemistry-Lectures and Lip2Wav-Chess-Analysis datasets. Our demo video can be found at https://glowlts.github.io/.

FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis

“…Griffin-Lim algorithm [11]). These works outperform previous works by a wide margin under unconstrained settings [15,24].…”

“…As this topic has only recently attracted attention of the researchers, there are not many works on it currently. Prajwal et al [24] firstly propose an autoregressive sequence-to-sequence model modified from Tacotron 2 [31] to tackle this problem, which generates mel-spectrograms conditioned on video frames; He et al [15] use a non-autoregressive architecture to accelerate inference and use a Glow [19] module for mel-spectrogram refinement.…”

“…Current works [15,24] show that the sequence-to-sequence [4,32] architecture is an effective solution to this problem. These works combine a visual encoder consisting of a stack of 3D-CNNs [35] and an LSTM [28] with acoustic models from TTS models [25,26,31], either adopting an autoregressive architecture [24] or a flow-based non-autoregressive architecture [15].…”

“…Current works [15,24] show that the sequence-to-sequence [4,32] architecture is an effective solution to this problem. These works combine a visual encoder consisting of a stack of 3D-CNNs [35] and an LSTM [28] with acoustic models from TTS models [25,26,31], either adopting an autoregressive architecture [24] or a flow-based non-autoregressive architecture [15]. They use a twostage pipeline that first generates mel-spectrograms as intermediate representations, and then synthesizes audio waveforms from the spectrograms with a signal-processing-based algorithm (i.e.…”

“…In addition, both the autoregressive model [24] and the flowbased non-autoregressive model [15] suffer from inefficiencies in either inference time or memory usage due to their model characteristics. The autoregressive model suffers from high inference latency due to its recursive nature [12,13,26], while the flow-based model has a large amount of parameters which causes high memory occupancy (see Section 5.6).…”