Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech

Yang, Geng; Yang, Shan; Li, Kai; Fang, Peng; Chen, Wei; Xie, Lei

doi:10.1109/slt48900.2021.9383551

Cited by 132 publications

(77 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human hearing is highly sensitive to irregularities and discontinuities in the periodic nature of any audio [9], which makes it even harder for any NN to generate human-like speech audio. However, in recent years, researchers have achieved extraordinary success using neural networks for generating speech audio from text [15]. Most of these successes are dominated by the autoregressive models.…”

Section: A Audio Generationmentioning

confidence: 99%

Guided Generative Adversarial Neural Network for Representation Learning and Audio Generation Using Fewer Labelled Audio Data

Haque

Rana

Liu

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

The Generation power of Generative Adversarial Neural Networks (GANs) has shown great promise to learn representations from unlabelled data while guided by a small amount of labelled data. We aim to utilise the generation power of GANs to learn Audio Representations. Most existing studies are, however, focused on images. Some studies use GANs for speech generation, but they are conditioned on text or acoustic features, limiting their use for other audio, such as instruments, and even for speech where transcripts are limited. This paper proposes a novel GAN-based model that we named Guided Generative Adversarial Neural Network (GGAN), which can learn powerful representations and generate good-quality samples using a small amount of labelled data as guidance. Experimental results based on a speech [Speech Command Dataset (S09)] and a non-speech [Musical Instrument Sound dataset (Nsyth)] dataset demonstrate that using only 5% of labelled data as guidance, GGAN learns significantly better representations than the state-of-the-art models.

show abstract

Section: A Audio Generationmentioning

confidence: 99%

Guided Generative Adversarial Neural Network for Representation Learning and Audio Generation Using Fewer Labelled Audio Data

Haque

Rana

Liu

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…The authors of [20]- [22] incorporated a flow-based generative model based on Glow [42], that can be directly learned by minimizing the negative log-likelihood of data without a distillation process. Another group of non-AR models [25]- [29] is based on an adversarial training framework [43]. Combining adversarial loss and auxiliary loss, such as multi-resolution short-time Fourier transform (STFT) loss and feature matching loss, enables non-AR models to be learned efficiently.…”

Section: Neural Waveform Generative Modelsmentioning

confidence: 99%

PeriodNet: A Non-Autoregressive Raw Waveform Generative Model With a Structure Separating Periodic and Aperiodic Components

et al. 2021

View full text Add to dashboard Cite

This paper presents PeriodNet, a non-autoregressive (non-AR) waveform generative model with a new model structure for modeling periodic and aperiodic components in speech waveforms. Non-AR raw waveform generative models have enabled the fast generation of high-quality waveforms. However, the variations of waveforms that these models can reconstruct are limited by training data. In addition, typical non-AR models reconstruct a speech waveform from a single Gaussian input despite the mixture of periodic and aperiodic signals in speech. These may significantly affect the waveform generation process in some applications such as singing voice synthesis systems, which require reproducing accurate pitch and natural sounds with less periodicity, including husky and breath sounds. PeriodNet uses a parallel or series model structure to model a speech waveform to tackle these problems. Two sub-generators connected in parallel or in series take an explicit periodic and aperiodic signal (sine wave and Gaussian noise) as an input. Since PeriodNet models periodic and aperiodic components by focusing on whether these input signals are autocorrelated or not, it does not require external periodic/aperiodic decomposition during training. Experimental results show that our proposed structure improves the naturalness of generated waveforms. We also show that speech waveforms with a pitch outside of the training data range can be generated with more naturalness.

show abstract

“…However, due to the high computational cost and timeconsumed generation, real-time applications could become challenges. Researchers have adopted multi-band generation such as Multiband-WaveRNN [42], Multiband-MelGAN [40] to speed up waveform modeling. Related multi-band vocoders generate each sub-band of waveform, and then conduct bands splicing using Pseudo Quadrature Mirror Filter Bank (PQMF) [24].…”

Section: Multi-band Generartionmentioning

confidence: 99%

“…Further, although several singing voice datasets such as MIR-1K dataset [15] and JukeBox [7] have been released for research purposes, but the corpora are not so large as expected for multiple tasks. 2) Several parallel generation methods [40,42] have been proposed to speed up waveforms synthesis. However, existing multi-band architectures do not consider characteristic differences among frequency bands, so a powerful frequency-adapted multi-band technique is required.…”

Section: Introductionmentioning

confidence: 99%

Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus

Huang

Chen

Ren

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost. Existing open corpora could not meet requirements for high-fidelity singing voice synthesis because of the scale and quality weaknesses. Previous vocoders have difficulty in multi-singer modeling, and a distinct degradation emerges when conducting unseen singer singing voice generation. To accelerate singing voice researches in the community, we release a large-scale, multi-singer Chinese singing voice dataset OpenSinger. To tackle the difficulty in unseen singer modeling, we propose Multi-Singer, a fast multi-singer vocoder with generative adversarial networks. Specifically, 1) Multi-Singer uses a multi-band generator to speed up both training and inference procedure. 2) to capture and rebuild singer identity from the acoustic feature (i.e., mel-spectrogram), Multi-Singer adopts a singer conditional discriminator and conditional adversarial training objective.3) to supervise the reconstruction of singer identity in the spectrum envelopes in frequency domain, we propose an auxiliary singer perceptual loss. The joint training approach effectively works in GANs for multi-singer voices modeling. Experimental results verify the effectiveness of OpenSinger and show that Multi-Singer improves unseen singer singing voices modeling in both speed and quality over previous methods. The further experiment proves that combined with FastSpeech 2 as the acoustic model, Multi-Singer achieves strong robustness in the multi-singer singing voice synthesis pipeline. Samples are available at https://Multi-Singer.github.io/ CCS CONCEPTS• Applied computing → Sound and music computing; • Computing methodologies → Natural language generation.

show abstract

Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech

Cited by 132 publications

References 23 publications

Guided Generative Adversarial Neural Network for Representation Learning and Audio Generation Using Fewer Labelled Audio Data

Guided Generative Adversarial Neural Network for Representation Learning and Audio Generation Using Fewer Labelled Audio Data

PeriodNet: A Non-Autoregressive Raw Waveform Generative Model With a Structure Separating Periodic and Aperiodic Components

Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus

Contact Info

Product

Resources

About