Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

Simon, Welker,; Richter, Julius; Gerkmann, Timo

doi:10.48550/arxiv.2203.17004

Cited by 3 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimental results show that SGMSE [124] maintains more natural structures with fewer artifacts than prior work, and achieves quantitative improvements, e.g., an SI-SAR improvement of 3dB over CDiffuSE [63].…”

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 92%

“…Although formulated on diffusion models, CDiffuSE [63] is trained to estimate the difference between clean and noisy speech. Therefore, it is pointed out in [124] that CDiffuSE [63] can be considered a discriminative task. To make the method pure generative and also avoid any noise prior, SGMSE [124] proposes a method based on stochastic differential equations (SDE) [103,105].…”

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 99%

“…Therefore, it is pointed out in [124] that CDiffuSE [63] can be considered a discriminative task. To make the method pure generative and also avoid any noise prior, SGMSE [124] proposes a method based on stochastic differential equations (SDE) [103,105]. The score function is used to evaluate the quality of the enhanced signal and guide the optimization of the model.…”

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 99%

“…The score function is used to evaluate the quality of the enhanced signal and guide the optimization of the model. In contrast to DiffuSE [62] and CDiffuSE [63] performing diffusion on the time-domain waveform, SGMSE [124] investigates speech enhancement on the time-frequency shorttime Fourier transform (STFT) domain to exploit its rich structures. Beyond amplitude of the complex coefficients, it also works on directly enhancing the phase so that an inverse STFT can be applied without extra need of phase retrieval.…”

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 99%

See 3 more Smart Citations

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

Zhang¹,

Zhang²,

Zheng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction.With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys that either lack the recent progress of diffusion-based speech synthesis or highlight an overall picture of applying diffusion model in multiple fields. Specifically, this work first briefly introduces the background of audio and diffusion model. As for the text-to-speech task, we divide the methods into three categories based on the stage where diffusion model is adopted: acoustic model, vocoder and end-to-end framework. Moreover, we categorize various speech enhancement tasks by either certain signals are removed or added into the input speech. Comparisons of experimental results and discussions are also covered in this survey.

show abstract

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 92%

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 99%

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 99%

Section: Audio Restoration In the Time-frequency Domainmentioning

confidence: 99%

See 2 more Smart Citations

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

Zhang¹,

Zhang²,

Zheng³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Denoising diffusion probabilistic models (diffusion models for short) have achieved the state-of-theart (SOTA) generation results in various tasks, including image [34,22,8,7,33,39,44] and super resolution image generation [13,31,41,25], text-to-image generation [23,11,14,28], text-to-speech synthesis [4,15,27,17,16,5] and speech enhancement [20,21,42]. Especially, in audio synthesis, diffusion models have shown strong ability in modelling both spectrogram features [27,17] and raw waveforms [4,15,5].…”

Section: Denoising Diffusion Probabilistic Modelsmentioning

confidence: 99%

BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis

Leng¹,

Chen²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Binaural audio plays a significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasing attention. This synthesis process involves not only the basic physical warping of the mono audio, but also room reverberations and head/ear related filtrations, which, however, are difficult to accurately simulate in traditional digital signal processing. In this paper, we formulate the synthesis process from a different perspective by decomposing the binaural audio into a common part that shared by the left and right channels as well as a specific part that differs in each channel. Accordingly, we propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Specifically, in the first stage, the common information of the binaural audio is generated with a single-channel diffusion model conditioned on the mono audio, based on which the binaural audio is generated by a two-channel diffusion model in the second stage. Combining this novel perspective of two-stage synthesis with advanced generative models (i.e., the diffusion models), the proposed BinauralGrad is able to generate accurate and high-fidelity binaural audio samples. Experiment results show that on a benchmark dataset, BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics (Wave L2: 0.128 vs. 0.157, MOS: 3.80 vs. 3.61). The generated audio samples are available online 3 .

show abstract

A Survey of Audio Enhancement Algorithms for Music, Speech, Bioacoustics, Biomedical, Industrial, and Environmental Sounds by Image U-Net

Gul,

Khan

2023

IEEE Access

View full text Add to dashboard Cite

The recent surge in the use of Deep Neural Networks (DNNs) has also made its mark in the field of Audio Enhancement (AE), providing much better quality than the classical methods. Although, there are dedicated audio processing DNNs, yet, many recent models of AE have utilized U-Net: a DNN based on Convolutional Neural Network (CNN), fundamentally developed for image segmentation. It is found that the useful features hidden in the time domain are highlighted when the audio signal is converted to a spectrogram, which can be treated as an image. In this article, we will review the recent work, utilizing U-Nets for different AE applications. Different than other published reviews, this review focuses entirely on AE techniques based on image U-Nets. We will discuss the need for AE, U-Net comparison to other DNNs, the benefits of converting the audio to 2D, input representations that are useful for different AE applications, the architecture of vanilla U-Net and the pre-trained models, variations in vanilla architecture incorporated in different E models, and the state-of-the-art AE algorithms based on U-Net in various applications. Apart from speech and music, this article discusses a wide range of audio signals e.g. environmental, biomedical, bioacoustics, and industrial sounds, not covered collectively in a single article in previously published studies. The article ends with the discussion of colored spectrograms in future AE applications.INDEX TERMS CNNs; image processing deep neural networks; pre-trained networks; spectrogram.;U-Net.

show abstract

Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

Cited by 3 publications

References 18 publications

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis

A Survey of Audio Enhancement Algorithms for Music, Speech, Bioacoustics, Biomedical, Industrial, and Environmental Sounds by Image U-Net

Contact Info

Product

Resources

About