Revisiting Over-Smoothness in Text to Speech

Abstract

Non-autoregressive text to speech (NAR-TTS) models have attracted much attention from both academia and industry due to their fast generation speed. One limitation of NAR-TTS models is that they ignore the correlation in time and frequency domains while generating speech mel-spectrograms, and thus cause blurry and over-smoothed results. In this work, we revisit this over-smoothing problem from a novel perspective: the degree of over-smoothness is determined by the gap between the complexity of data distributions and the capability of modeling methods. Both simplifying data distributions and improving modeling methods can alleviate the problem. Accordingly, we first study methods reducing the complexity of data distributions. Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods. Based on these studies, we find that 1) methods that provide additional condition inputs reduce the complexity of data distributions to model, thus alleviating the over-smoothing problem and achieving better voice quality. 2) Among advanced modeling methods, Laplacian mixture loss performs well at modeling multimodal distributions and enjoys its simplicity, while GAN and Glow achieve the best voice quality while suffering from increased training or model complexity.3) The two categories of methods can be combined to further alleviate the over-smoothness and improve the voice quality. 4) Our experiments on the multi-speaker dataset lead to similar conclusions as above and providing more variance information can reduce the difficulty of modeling the target data distribution and alleviate the requirements for model capacity.

A Deliberation-Based Joint Acoustic and Text Decoder

“…There have been multiple approaches for augmenting E2E models and training procedures to incorporate unpaired text data. Broadly speaking, these approaches use some combination of an LM trained on text data (shallow, cold, deep fusion [10,11,12,13]) and a multi-stage training procedure that incorporates unpaired data ("weak distillation" [14], "backtranslation" [15], "cycle-consistency" [16,17,18]). Each approach produces improvements in performance, but also increases some combination of model size, training and inference complexity, making it less desirable for on-device applications.…”

Let There Be Sound: Reconstructing High Quality Speech from Silent Videos

“…During inference stage, we take samples z from the prior distribution and feed them into the post-net reversely to generate the final mel-spectrogram. As proved in (Ren et al 2022), this flow-based module enhances the capability of modelling complex data distributions, which helps to address one-tomany mapping problem.…”

“…where Y j denotes the j th frame of the ground truth melspectrogram with length T m , and Ŷj stands for the j th frame of predicted mel-spectrogram. Note that the variance predictors simplify the acoustic target distribution by providing conditional information, thereby mitigating the one-to-many mapping issue (Ren et al 2022). We analyse the effect of variance information in our experiment section.…”

“…With the help of the acoustic variations, the model can not only ease the one-to-many mapping but also learn prosody of speech which is a key factor for realistic speech synthesis (Skerry-Ryan et al 2018;Sun et al 2020). To further address the one-to-many mapping, we use a flow based post-net which refines acoustic representations with enhanced modelling capability of capturing fine-grained details (Ren et al 2022). Combined with the variance information, the post-net helps to learn the complex one-to-many-mapping between phonemes and speech, thereby improving the naturalness of the synthesised speech.…”

Improving Multi-Speaker TTS Prosody Variance with a Residual Encoder and Normalizing Flows

“…However, there are still many challenges and opportunities in this domain [11], particularly when it comes to exploiting large amounts of data. On the speech generation side, one of the main difficulties is to build a model that correctly aligns the phonetic and acoustic sequences, leading to a natural prosody with fluent speech and high intelligibility, while still cap-turing the prosody variations [25]. On the opposite side, automatic speech recognition systems struggle with long-tail words recognition [35], and speech vs background disentanglement [18].…”