FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Abstract

Advanced text to speech (TTS) models such as FastSpeech [20] can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-tomany mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacherstudent distillation pipeline is complicated, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs during training and use predicted values during inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of full end-to-end training and even faster inference than FastSpeech. Experimental results show that 1) FastSpeech 2 and 2s outperform FastSpeech in voice quality with much simplified training pipeline and reduced training time; 2) FastSpeech 2 and 2s can match the voice quality of autoregressive models while enjoying much faster inference speed. Audio samples are available at https://speechresearch. github.io/fastspeech2/.

Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis

“…These results indicate the effectiveness of Conformer blocks and Mel-based adversarial training method on the singing voice synthesis. From the generated results, baseline models showed limited capability in handling the pitch that falls in a long tail distribution in terms of pitch frequency in the training set 5 . How to deal with this issue raised by the long tail distribution and even to perform a song with note pitch that is beyond the training set pitch distribution should be considered in future work.…”

“…A typical two-stage singing voice synthesis framework, which consists of an acoustic model and a vocoder is adopted in the experiments. In practice, Fastspeech2 [5] and HiFi-GAN [11], which are two popular models for spectrogram synthesis and waveform reconstruction respectively, are utilized in this work.…”

“…The development of neural end-to-end text-to-speech (TTS) models [1,2,3,4,5,6] has greatly promoted speech synthesis. Generally, with a well-trained neural acoustic model [2,5,6,7] and a neural vocoder [8,9,10,11], or alternatively using fully end-to-end models [12,13,14] which directly construct wave signals from text input, it is able to synthesize high-quality neutral speech. Recently, much attention has been attracted to synthesizing expressive speech, such as stylized speech [15,16], emotional speech [17,18,19,20,21,22], and also singing voice [23,24].…”

Audio representations for deep learning in sound synthesis: A review

“…Apart from the original spectrogram, deep learning architectures have also experimented with non linear spectrograms such as mel-spectrograms [21] [22] [23] [24] [25] [26] [27] or Constant-Q Transformations (CQT) [28]. The mel-spectrogram is generated by the application of perceptual filters on the DFT called mel-filter bands.…”

Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus

“…We have attached the modified FastSpeech 2 in appendix F in the supplementary materials. During training, the configuration follows prior work [31]. Since the F0 and duration are usually known in singing voice synthesis, we remove the Table 7: The MOS results with 95% confidence intervals on each Singing voice synthesis system.…”