DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

Liu, Songxiang; Su, Dan; Yu, Dong

doi:10.48550/arxiv.2201.11972

Cited by 6 publications

(16 citation statements)

References 16 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The development of text-to-speech has undergone through a shift from a three-stage framework to a two-stage framework, as shown in Figure 1. Before applying neural networks, Diff-TTS [31] Grad-TTS [85] Code / Project Efficient acoustic model ProDiff [28] Project DiffGAN-TTS [60] Code Adaptive multi-speaker model Grad-TTS with ILVR [52] Project Grad-StyleSpeech [32] Project Guided-TTS [37] Guided-TTS 2 [38] Project With discrete latent space Diffsound [128] Project NoreSpeech [127] Fine-grained control EmoDiff [22] Project Vocoder…”

Section: Text-to-speech Synthesismentioning

confidence: 99%

“…Prior work [126] attributes the thousands of denoising steps in diffusion models to the fact that they commonly approximate the denoising distribution with Gaussian noise, thus requiring small step size. Inspired by [126] that adopts GAN to model the denoising distribution which enables larger step size and less steps, DiffGAN-TTS [60] applies a pretrained GAN as the acoustic generator for acceleration. To further speed up, DiffGAN-TTS [60] also introduces an active shallow diffusion mechanism that conduct denoising conditioned on the coarse prediction by pretrained GAN.…”

Section: Towards Efficient Acoustic Modelmentioning

confidence: 99%

“…Inspired by [126] that adopts GAN to model the denoising distribution which enables larger step size and less steps, DiffGAN-TTS [60] applies a pretrained GAN as the acoustic generator for acceleration. To further speed up, DiffGAN-TTS [60] also introduces an active shallow diffusion mechanism that conduct denoising conditioned on the coarse prediction by pretrained GAN. Experimental results show that DiffGAN-TTS [60] can generate high-quality audio with only 1 step.…”

Section: Towards Efficient Acoustic Modelmentioning

confidence: 99%

“…To further speed up, DiffGAN-TTS [60] also introduces an active shallow diffusion mechanism that conduct denoising conditioned on the coarse prediction by pretrained GAN. Experimental results show that DiffGAN-TTS [60] can generate high-quality audio with only 1 step.…”

Section: Towards Efficient Acoustic Modelmentioning

confidence: 99%

See 3 more Smart Citations

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

Zhang¹,

Zhang²,

Zheng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction.With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys that either lack the recent progress of diffusion-based speech synthesis or highlight an overall picture of applying diffusion model in multiple fields. Specifically, this work first briefly introduces the background of audio and diffusion model. As for the text-to-speech task, we divide the methods into three categories based on the stage where diffusion model is adopted: acoustic model, vocoder and end-to-end framework. Moreover, we categorize various speech enhancement tasks by either certain signals are removed or added into the input speech. Comparisons of experimental results and discussions are also covered in this survey.

show abstract

Section: Text-to-speech Synthesismentioning

confidence: 99%

Section: Towards Efficient Acoustic Modelmentioning

confidence: 99%

Section: Towards Efficient Acoustic Modelmentioning

confidence: 99%

Section: Towards Efficient Acoustic Modelmentioning

confidence: 99%

See 2 more Smart Citations

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

Zhang¹,

Zhang²,

Zheng³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Discriminator: Multi-scaled discriminators (MSD) [17] and Joint Conditional Unconditional discriminators (JCUD) [18] have been proven as the most efficient models in audio synthetic tasks. Inspired of them, we propose a Joint Conditional Unconditional Multi-scale discriminator (JCU-MSD), i.e., D φ , which is shown in Fig.…”

Section: Feature Reconstruction Networkmentioning

confidence: 99%

PSVRF: Learning to restore Pitch-Shifted Voice without reference

Li¹,

Lin²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

Pitch scaling algorithms have a significant impact on the security of Automatic Speaker Verification (ASV) systems. Although numerous anti-spoofing algorithms have been proposed to identify the pitch-shifted voice and even restore it to the original version, they either have poor performance or require the original voice as a reference, limiting the prospects of applications. In this paper, we propose a no-reference approach termed PSVRF 1 for high-quality restoration of pitchshifted voice. Experiments on AISHELL-1 and AISHELL-3 demonstrate that PSVRF can restore the voice disguised by various pitch-scaling techniques, which obviously enhances the robustness of ASV systems to pitch-scaling attacks. Furthermore, the performance of PSVRF even surpasses that of the state-of-the-art reference-based approach.

show abstract

MixGAN-TTS: Efficient and Stable Speech Synthesis Based on Diffusion Model

Deng

Qiu

et al. 2023

IEEE Access

View full text Add to dashboard Cite

This paper describes MixGAN-TTS, an efficient and stable non-autoregressive speech synthesis based on diffusion model. The MixGAN-TTS uses a linguistic encoder based on soft phoneme-level alignment and hard word-level alignment approach which explicitly extracts word-level semantic information, and introduces pitch and energy predictors to optimally predict the rhythmic information of the audio. Specifically, we use the GAN to replace the Gaussian function to model the denoising distribution, aiming to enlarge the denoising steps size and reduce the number of denoising steps to accelerate the sampling speed of diffusion model. Diffusion model using GAN can significantly reduce the denoising steps, and to some extent solve the problem of not being able to apply in real-time. The mel-spectrogram is converted into the final audio by the HiFi-GAN vocoder. Experimental results show that the MixGAN-TTS outperforms the other models compared in terms of audio quality and mel-spectrogram modeling capability for 4 denoising steps. The ablation studies demonstrate that the structure of MixGAN-TTS is effective.INDEX TERMS Speech synthesis, diffusion model, mixture attention mechanism, deep learning.

show abstract

DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

Cited by 6 publications

References 16 publications

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

PSVRF: Learning to restore Pitch-Shifted Voice without reference

MixGAN-TTS: Efficient and Stable Speech Synthesis Based on Diffusion Model

Contact Info

Product

Resources

About