Guided Variational Autoencoder for Speech Enhancement with a Supervised Classifier

Abstract

Recently, variational autoencoders have been successfully used to learn a probabilistic prior over speech signals, which is then used to perform speech enhancement. However, variational autoencoders are trained on clean speech only, which results in a limited ability of extracting the speech signal from noisy speech compared to supervised approaches. In this paper, we propose to guide the variational autoencoder with a supervised classifier separately trained on noisy speech. The estimated label is a high-level categorical variable describing the speech signal (e.g. speech activity) allowing for a more informed latent distribution compared to the standard variational autoencoder. We evaluate our method with different types of labels on real recordings of different noisy environments. Provided that the label better informs the latent distribution and that the classifier achieves good performance, the proposed approach outperforms the standard variational autoencoder and a conventional neural networkbased supervised approach.

Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement

“…The VAE can be conditioned on a label yn ∈ Y describing a speech attribute (e.g. speech activity) that allows for a more explicit control of speech generation [13]. A common approach is to make use of the label yn by directly inputting it in both the encoder E φ,z (|sn| 2 , yn) and the decoder D θ (zn, yn) (see Fig 1b) [11,12,13].…”

“…tion [10]. For various speech-related tasks, VAEs have been conditioned on a label describing a speech attribute, such as speaker identity [11,12], phoneme [12] or speech activity [13]. Ideally, the label should be independent from the other latent dimensions to obtain an explicit control of speech generation.…”

Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders

“…This approach is computationally efficient since it does not require sampling or gradient descent at each step of the algorithm. More recently, a guided VAE was proposed in [36], where the VAE-based clean speech signal prior is defined conditionally on the voice activity detection or the ideal binary mask. This guiding information has to be provided by a supervised classifier, separately trained on noisy speech signals.…”

“…where Σ θs,t = Σ θs,t (s 1:t−1 , z1:t ) and z1:t is sampled from p θ (z 1:t |x 1:T ). In practice, this posterior distribution is also intractable, we thus propose a variational approximation q φ (z 1:T |x 1:T ) whose parameters φ need to be jointly estimated together with the noisy mixture model parameters ϕ, in order to compute the speech estimate in (36). As detailed in the next section, we propose a VEM algorithm to do that.…”

“…Compute the clean speech signal estimate ŝt for t = 1 to T using (36) ensured provided that these parameters are initialized with non-negative values.…”