Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

Simon, Welker,; Richter, Julius; Gerkmann, Timo

doi:10.21437/interspeech.2022-10653

Cited by 36 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most similar to our proposed IR-SDE is the work of (Welker et al, 2022b;Richter et al, 2022), in which a mean-reverting SDE is applied to the speech processing tasks of speech enhancement and speech dereverberation. They utilize an SDE of the form (3) but with a different 𝜎 𝑡 and a constant 𝜃 , i.e.…”

Section: Related Workmentioning

confidence: 99%

“…As shown in Section 5.3, both of these are outperformed by our cosine 𝜃 scheduler. Moreover, while (Welker et al, 2022b;Richter et al, 2022;Welker et al, 2022a) all use the standard score matching objective, we introduce an alternative maximum likelihood-based loss function that stabilizes training and leads to improved restoration performance. Finally, we demonstrate the general applicability of our approach by applying it to six diverse image restoration tasks.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Image Restoration with Mean-Reverting Stochastic Differential Equations

Luo¹,

Gustafsson²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

This paper presents a stochastic differential equation (SDE) approach for general-purpose image restoration. The key construction consists in a mean-reverting SDE that transforms a highquality image into a degraded counterpart as a mean state with fixed Gaussian noise. Then, by simulating the corresponding reverse-time SDE, we are able to restore the origin of the low-quality image without relying on any taskspecific prior knowledge. Crucially, the proposed mean-reverting SDE has a closed-form solution, allowing us to compute the ground truth timedependent score and learn it with a neural network. Moreover, we propose a maximum likelihood objective to learn an optimal reverse trajectory which stabilizes the training and improves the restoration results. In the experiments, we show that our proposed method achieves highly competitive performance in quantitative comparisons on image deraining, deblurring, and denoising, setting a new state-of-the-art on two deraining datasets. Finally, the general applicability of our approach is further demonstrated via qualitative results on image super-resolution, inpainting, and dehazing. Code is available at https://github.com/ Algolzw/image-restoration-sde.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Image Restoration with Mean-Reverting Stochastic Differential Equations

Luo¹,

Gustafsson²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This research area can be categorized into two groups: conditional methods, which require specialized training for specific problems, and zero-shot methods, which leverage priors from unconditional diffusion models. Within the category of conditional models, several works target speech enhancement [33]- [36], image deblurring [8], and JPEG reconstruction [37], among others. It may be noted that these methods all require pairs of clean/degraded samples and a well-thought training data pipeline.…”

Section: B Diffusion Models For Blind Inverse Problemsmentioning

confidence: 99%

BEHM-GAN: Bandwidth Extension of Historical Music Using Generative Adversarial Networks

Moliner

Välimäki

2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Audio bandwidth extension aims to expand the spectrum of bandlimited audio signals. Although this topic has been broadly studied during recent years, the particular problem of extending the bandwidth of historical music recordings remains an open challenge. This paper proposes a method for the bandwidth extension of historical music using generative adversarial networks (BEHM-GAN) as a practical solution to this problem. The proposed method works with the complex spectrogram representation of audio and, thanks to a dedicated regularization strategy, can effectively extend the bandwidth of out-of-distribution real historical recordings. The BEHM-GAN is designed to be applied as a second step after denoising the recording to suppress any additive disturbances, such as clicks and background noise. We train and evaluate the method using solo piano classical music. The proposed method outperforms the compared baselines in both objective and subjective experiments. The results of a formal blind listening test show that BEHM-GAN significantly increases the perceptual sound quality in early-20th-century gramophone recordings. For several items, there is a substantial improvement in the mean opinion score after enhancing historical recordings with the proposed bandwidthextension algorithm. This study represents a relevant step toward data-driven music restoration in real-world scenarios.

show abstract

“…k and l denote the frequency bin and time frame indexes. Note that the existing diffusion-based models can be extended to complex-valued data, such as STFT coefficients [24,25].…”

Section: Diffusion-based Generative Model On Clean Speech Datamentioning

confidence: 99%

A Versatile Diffusion-based Generative Refiner for Speech Enhancement

Sawata¹,

Murata²,

Takida³

et al. 2022

Preprint

View full text Add to dashboard Cite

Although deep neural network (DNN)-based speech enhancement (SE) methods outperform the previous non-DNN-based ones, they often degrade the perceptual quality of generated outputs. To tackle this problem, We introduce a DNN-based generative refiner aiming to improve perceptual speech quality pre-processed by an SE method. As the refiner, we train a diffusion-based generative model by utilizing a dataset consisting of clean speech only. Then, the model replaces the degraded and distorted parts caused by a preceding SE method with newly generated clean parts by denoising diffusion restoration. Once our refiner is trained on a set of clean speech, it can be applied to various SE methods without additional training specialized for each SE module. Therefore, our refiner can be a versatile post-processing module w.r.t. SE methods and has high potential in terms of modularity. Experimental results show that our method improved perceptual speech quality regardless of the preceding SE methods used.

show abstract

Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

Cited by 36 publications

References 0 publications

Image Restoration with Mean-Reverting Stochastic Differential Equations

Image Restoration with Mean-Reverting Stochastic Differential Equations

BEHM-GAN: Bandwidth Extension of Historical Music Using Generative Adversarial Networks

A Versatile Diffusion-based Generative Refiner for Speech Enhancement

Contact Info

Product

Resources

About