Switching Variational Auto-Encoders for Noise-Agnostic Audio-Visual Speech Enhancement

Abstract

Recently, audio-visual speech enhancement has been tackled in the unsupervised settings based on variational autoencoders (VAEs), where during training only clean data is used to train a generative model for speech, which at test time is combined with a noise model, e.g. nonnegative matrix factorization (NMF), whose parameters are learned without supervision. Consequently, the proposed model is agnostic to the noise type. When visual data are clean, audio-visual VAE-based architectures usually outperform the audio-only counterpart. The opposite happens when the visual data are corrupted by clutter, e.g. the speaker not facing the camera. In this paper, we propose to find the optimal combination of these two architectures through time. More precisely, we introduce the use of a latent sequential variable with Markovian dependencies to switch between different VAE architectures through time in an unsupervised manner: leading to switching variational auto-encoder (SwVAE). We propose a variational factorization to approximate the computationally intractable posterior distribution. We also derive the corresponding variational expectation-maximization algorithm to estimate the parameters of the model and enhance the speech signal. Our experiments demonstrate the promising performance of SwVAE.

Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders

“…In all the following, we do not consider SRNN-TF-Pred anymore, since this model was shown to be inefficient for speech enhancement. The circumflex accent shape of the curves is often encountered when reporting improvement over the noisy input [28], [60]- [62]. Indeed, when the input is very noisy, it is difficult to improve the quality.…”

Audio-Visual Wake Word Spotting System For MISP Challenge 2021

“…(AV) multi-modal has been applied widely in speech community [6][7][8][9][10][11][12]. The visual information obtained by analyzing lip shapes or facial expressions of the visual modality is more robust than the audio information from complex scenarios.…”