Speech Signal Improvement Using Causal Generative Diffusion Models

Richter, Julius; Simon, Welker,; Jean-Marie, Lemercier,; Lay, Bunlong; Peer, Tal; Gerkmann, Timo

doi:10.1109/icassp49357.2023.10095126

Cited by 4 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the AudioSet pretraining outperformed the Lib-riTTS one in all variations, which implies that the diverse audio sources and recording environments in AudioSet have been beneficial for generalization. SGMSE+ is reported to have SOTA generalization ability [32], and while it worked similarly to the pretrained ViT-AE in our study, it does not work for classification tasks.…”

Section: Speech Enhancementmentioning

confidence: 67%

Diversification Strategy of Sony Group

Choi¹

2021

KRCM

View full text Add to dashboard Cite

Audio classification and restoration are among major downstream tasks in audio signal processing. However, restoration derives less of a benefit from pretrained models compared to the overwhelming success of pretrained models in classification tasks. Due to such unbalanced benefits, there has been rising interest in how to improve the performance of pretrained models for restoration tasks such as speech enhancement (SE). Previous works have shown that the features extracted by pretrained audio encoders are effective for SE tasks, but these speech-specific encoder-only models usually require extra decoders to become compatible with SE tasks, and involve complicated pretraining procedures or complex data augmentation. Therefore, in pursuit of a universal audio model, the audio masked autoencoder (MAE) whose backbone is the autoencoder of Vision Transformers (ViT-AE), is extended from audio classification toward restoration tasks in this paper. ViT-AE naturally learns mel-to-mel mapping that is compatible with restoration tasks during pretraining. Among many restoration tasks, SE is chosen due to its wellestablished evaluation metrics and test data. We propose variations of ViT-AE to improve the SE performance, where the mel-to-mel variations yield high scores for non-intrusive metrics and the STFToriented variation is effective at standard intrusive metrics such as PESQ. Different variations can be used in accordance with the scenarios. Comprehensive evaluations and ablation studies show that MAE pretraining is also beneficial to SE tasks and help the ViT-AE to better generalize to out-of-domain distortions. We further found that large-scale noisy data of general audio sources, rather than clean speech, is sufficiently effective for pretraining.

show abstract

Section: Speech Enhancementmentioning

confidence: 67%

Diversification Strategy of Sony Group

Choi¹

2021

KRCM

View full text Add to dashboard Cite

show abstract

“…Legends-tencent [35] AGC → GSM-GAN (Restore) → Enhance Genius-team [36] TRGAN (Restore) → MTFAA-Lite (Enhance) Hitot [37] Half temporal, half frequency attention U-Net N&B [38] GateDCCRN (Repairing) → GateDCCRN, S-DCCRN (Denoising) Hamburg [39] Generative diffusion model (modified NCSN++) tencent [35] and Genius-team [36] perform particularly well. • There is still significant room for improvement in this test set for OVRL and SIG.…”

Section: Team Modelmentioning

confidence: 99%

ICASSP 2023 Speech Signal Improvement Challenge

Cutler,

Saabas,

Naderi

et al. 2024

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

The ICASSP 2023 Speech Signal Improvement Challenge is intended to stimulate research in the area of improving the speech signal quality in communication systems. The speech signal quality can be measured with SIG in ITU-T P.835 and is still a top issue in audio communication and conferencing systems. For example, in the ICASSP 2022 Deep Noise Suppression challenge, the improvement in the background and overall quality is impressive, but the improvement in the speech signal is not statistically significant. To improve the speech signal the following speech impairment areas must be addressed: coloration, discontinuity, loudness, reverberation, and noise. A training and test set was provided for the challenge, and the winners were determined using an extended crowdsourced implementation of ITU-T P.804's listening phase. The results show significant improvement was made across all measured dimensions of speech quality.

show abstract