Towards Accurate Lip-to-Speech Synthesis in-the-Wild

Abstract

Figure 1: We propose a novel approach for multi-speaker lip-to-speech synthesis in the wild. Prior works try to learn a language model directly from raw speech, which only provides weak supervision due to the presence of other acoustic variations such as voice, accents, and prosody. We solve this problem by relying on recent advancements in lip-to-text generation. We condition on the noisy text outputs and lip video to generate natural speech with clearly pronounced words.