Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement

Carbajal, Guillaume; Richter, Julius; Gerkmann, Timo

doi:10.1109/waspaa52581.2021.9632676

Cited by 13 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, this method can adopt various DNN structures [2], so the DNN-based SE algorithms [2] can be directly optimized by PVAE. This is not achieved by VAE-NMFbased algorithms [17]- [22]. The experimental results [26] indicate that the SE performance of the traditional DNN-based methods can be improved by introducing this PVAE-based DRL algorithm.…”

Section: Introductionmentioning

confidence: 92%

“…Recently, to improve traditional DNN's generalization ability, DRL-based SE algorithms are proposed [17]- [22]. The basic idea of these methods is that they use a variational autoencoder (VAE) [23] to learn speech representations when modeling speech, and apply a non-negative matrix factorization (NMF) to model noise.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, VAE is suitable for various speech generative tasks [23]- [25]. These VAE-based algorithms can effectively improve DNN's generalization ability, but they have difficulty obtaining good speech representations from the observed signal because they cannot disentangle speech representations from other latent representations [15], [17]- [22]. This causes the need to use a linear NMF to model noise, so their noise modeling ability is limited compared with these non-linear DNN-based methods [23].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A deep representation learning speech enhancement method using $β$-VAE

Xiang¹,

Højvang²,

Rasmussen³

et al. 2022

Preprint

View full text Add to dashboard Cite

In previous work, we proposed a variational autoencoder-based (VAE) Bayesian permutation training speech enhancement (SE) method (PVAE) which indicated that the SE performance of the traditional deep neural network-based (DNN) method could be improved by deep representation learning (DRL). Based on our previous work, we in this paper propose to use β-VAE to further improve PVAE's ability of representation learning. More specifically, our β-VAE can improve PVAE's capacity of disentangling different latent variables from the observed signal without the trade-off problem between disentanglement and signal reconstruction. This trade-off problem widely exists in previous β-VAE algorithms. Unlike the previous β-VAE algorithms, the proposed β-VAE strategy can also be used to optimize the DNN's structure. This means that the proposed method can not only improve PVAE's SE performance but also reduce the number of PVAE training parameters. The experimental results show that the proposed method can acquire better speech and noise latent representation than PVAE. Meanwhile, it also obtains a higher scale-invariant signal-todistortion ratio, speech quality, and speech intelligibility.

show abstract

Section: Introductionmentioning

confidence: 92%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A deep representation learning speech enhancement method using $β$-VAE

Xiang¹,

Højvang²,

Rasmussen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Instead of learning a direct mapping from noisy to clean speech, generative models aim to learn the distribution of clean speech as a prior for speech enhancement. Several approaches have utilized deep generative models for speech enhancement using generative adversarial networks (GANs) [4], variational autoencoders (VAEs) [5,6,7,8], flow-based models [9], and more recently denoising diffusion probabilistic models (DDPMs) [10,11]. The main principle of these approaches is to learn the inherent properties of clean speech, We acknowledge the support by DASHH (Data Science in Hamburg -HELMHOLTZ Graduate School for the Structure of Matter) with the Grant-No.…”

Section: Introductionmentioning

confidence: 99%

Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

Simon¹,

Richter²,

Gerkmann³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Score-based generative models (SGMs) have recently shown impressive results for difficult generative tasks such as the unconditional and conditional generation of natural images and audio signals. In this work, we extend these models to the complex short-time Fourier transform (STFT) domain, proposing a novel training task for speech enhancement using a complexvalued deep neural network. We derive this training task within the formalism of stochastic differential equations, thereby enabling the use of predictor-corrector samplers. We provide alternative formulations inspired by previous publications on using SGMs for speech enhancement, avoiding the need for any prior assumptions on the noise distribution and making the training task purely generative which, as we show, results in improved enhancement performance.

show abstract

“…This reparameterization trick also ensures that the parameter update in VAE is differentiable. At present, VAE has been widely used in the SE task [98,[105][106][107][108][109]. However, most of these algorithms only apply VAE to learn the clean speech representation, and they do not attempt to disentangle clean speech representation with other noise representations, which leads to inaccurate speech estimation.…”

Section: Variational Autoencoder (Vae)mentioning

confidence: 99%

Data-driven Speech Enhancement: from Non-negative Matrix Factorization to Deep Representation Learning

Xiang¹

View full text Add to dashboard Cite

In natural listening environments, speech signals are easily distorted by various acoustic interference, which reduces the speech quality and intelligibility of human listening; meanwhile, it makes difficult for many speech-related applications, such as automatic speech recognition (ASR). Thus, many speech enhancement (SE) algorithms have been developed in the past decades. However, most current SE algorithms are difficult to capture underlying speech information (e.g., phoneme) in the SE process. This causes it to be challenging to know what specific information is lost or interfered with in the SE process, which limits the application of enhanced speech. For instance, some SE algorithms aimed to improve human listening usually damage the ASR system.The objective of this dissertation is to develop SE algorithms that have the potential to capture various underlying speech representations (information) and improve the quality and intelligibility of noisy speech. This study starts by introducing the hidden Markov model (HMM) into the Non-negative Matrix Factorization (NMF) model (NMF-HMM) because HMM is a convenient way to find underlying speech information for better SE performance. The key idea is applying HMM to capture the underlying speech temporal dynamics information in the NMF model. Additionally, a computationally efficient method is also proposed to ensure that this NMF-HMM model can achieve fast online SE.Although NMF-HMM captures the underlying speech information, it is difficult to explain what detailed information is obtained. In addition, NMF-HMM cannot represent the underlying information in a vector form, which makes information analysis difficult. To address these problems, we introduce deep representation learning (DRL) for SE. DRL can also improve the SE performance of DNN-based algorithms since DRL can obtain a discriminative speech representation, which can reduce the requirements for the learning machine to perform a task successfully. Specifically, we propose a Bayesian permutation training variational autoencoder (PVAE) to analyze underlying speech information for SE, which can represent and disentangle underlying noisy speech information in a vector form. The experimental results indicate that disentangled signal representations can also help current DNN-based SE algorithms achieve better SE performance. Additionally, based on this PVAE framework, we propose applying β-VAE and generative adversarial networks to improve PVAE's information disentanglement and signal restoration ability, respectively.

show abstract

Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement

Cited by 13 publications

References 21 publications

A deep representation learning speech enhancement method using $β$-VAE

A deep representation learning speech enhancement method using $β$-VAE

Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

Data-driven Speech Enhancement: from Non-negative Matrix Factorization to Deep Representation Learning

Contact Info

Product

Resources

About