Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior

Sun, Guangzhi; Yu, Zhang; Weiss, Ron; Cao, Yuan; Zen, Heiga; Rosenberg, Andrew; Ramabhadran, Bhuvana; Wu, Yonghui

doi:10.1109/icassp40776.2020.9053436

Cited by 73 publications

(75 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…VQ-VAE [14] has been applied to various speech synthesis tasks, including diverse and controllable TTS [17,18], a new TTS framework based on symbol-to-symbol translation [19], speech coding [20], voice conversion [21], and representation learning [22,23,24,25].…”

Section: Vector Quantized Autoencoder For Speech Tasksmentioning

confidence: 99%

See 1 more Smart Citation

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

Yasuda

Wang

Yamagishd

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Explicit duration modeling is a key to achieving robust and efficient alignment in text-to-speech synthesis (TTS). We propose a new TTS framework using explicit duration modeling that incorporates duration as a discrete latent variable to TTS and enables joint optimization of whole modules from scratch. We formulate our method based on conditional VQ-VAE to handle discrete duration in a variational autoencoder and provide a theoretical explanation to justify our method. In our framework, a connectionist temporal classification (CTC) -based force aligner acts as the approximate posterior, and text-to-duration works as the prior in the variational autoencoder. We evaluated our proposed method with a listening test and compared it with other TTS methods based on soft-attention or explicit duration modeling. The results showed that our systems rated between soft-attention-based methods (Transformer-TTS, Tacotron2) and explicit duration modeling-based methods (Fastspeech).

show abstract

Section: Vector Quantized Autoencoder For Speech Tasksmentioning

confidence: 99%

“…Among the related methods, Sun et al applied conditional VQ-VAE to TTS, although their objective was diverse TTS for data augmentation rather than duration modeling [18]. Their method relies on soft-attention to align speech and phoneme.…”

Section: Vector Quantized Autoencoder For Speech Tasksmentioning

confidence: 99%

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

Yasuda

Wang

Yamagishd

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Another idea [33]- [37] is to extract the latent prosody embeddings to characterize prosody. Some [33]- [35] learn speech variations without explicit annotations for prosody or style.…”

Section: Tacotron-based Ttsmentioning

confidence: 99%

“…The learned prosody embeddings are usually not fully controllable and interpretable. Others [36], [37] just take the prosody embeddings as an auxiliary input to the TTS model.…”

Section: Tacotron-based Ttsmentioning

confidence: 99%

Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS

Liu

Şişman

Bao

et al. 2020

IEEE Signal Process. Lett.

View full text Add to dashboard Cite

Tacotron-based end-to-end speech synthesis has shown remarkable voice quality. However, the rendering of prosody in the synthesized speech remains to be improved, especially for long sentences, where prosodic phrasing errors can occur frequently. In this paper, we extend the Tacotron-based speech synthesis framework to explicitly model the prosodic phrase breaks. We propose a multi-task learning scheme for Tacotron training, that optimizes the system to predict both Mel spectrum and phrase breaks. To our best knowledge, this is the first implementation of multi-task learning for Tacotron based TTS with a prosodic phrasing model. Experiments show that our proposed training scheme consistently improves the voice quality for both Chinese and Mongolian systems.

show abstract

“…These systems are trained with well-articulated read speech. Attempts to model speech with various speaking styles have also been actively investigated in deep-learning-based speech synthesis studies [3]- [10].…”

Section: Introductionmentioning

confidence: 99%

Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences

et al. 2020

View full text Add to dashboard Cite

We have been investigating rakugo speech synthesis as a challenging example of speech synthesis that entertains audiences. Rakugo is a traditional Japanese form of verbal entertainment similar to a combination of one-person stand-up comedy and comic storytelling and is popular even today. In rakugo, a performer plays multiple characters, and conversations or dialogues between the characters make the story progress. To investigate how close the quality of synthesized rakugo speech can approach that of professionals' speech, we modeled rakugo speech using Tacotron 2, a state-of-the-art speech synthesis system that can produce speech that sounds as natural as human speech albeit under limited conditions, and an enhanced version of it with self-attention to better consider long-term dependencies. We also used global style tokens and manually labeled context features to enrich speaking styles. Through a listening test, we measured not only naturalness but also distinguishability of characters, understandability of the content, and the degree of entertainment. Although we found that the speech synthesis models could not yet reach the professional level, the results of the listening test provided interesting insights: 1) we should not focus only on the naturalness of synthesized speech but also the distinguishability of characters and the understandability of the content to further entertain audiences; 2) the fundamental frequency (f o ) expressions of synthesized speech are poorer than those of human speech, and more entertaining speech should have richer f o expression. Although there is room for improvement, we believe this is an important stepping stone toward achieving entertaining speech synthesis at the professional level.

show abstract

Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior

Cited by 73 publications

References 16 publications

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS

Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences

Contact Info

Product

Resources

About