Fine-Grained Robust Prosody Transfer for Single-Speaker Neural Text-To-Speech

Klimkov, Viacheslav; Ronanki, Srikanth; Rohnke, Jonas; Drugman, Thomas

doi:10.21437/interspeech.2019-2571

Cited by 60 publications

(34 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2. The acoustic encoder in the CTC model consists of convolution layers followed by a bidirectional LSTM, which is similar to that in [28]. The LatentNet φ in the prior consists of a CBHG-based linguistic encoder [29] and an additional LSTM layer.…”

Section: Methodsmentioning

confidence: 99%

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

Yasuda

Wang

Yamagishd

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Explicit duration modeling is a key to achieving robust and efficient alignment in text-to-speech synthesis (TTS). We propose a new TTS framework using explicit duration modeling that incorporates duration as a discrete latent variable to TTS and enables joint optimization of whole modules from scratch. We formulate our method based on conditional VQ-VAE to handle discrete duration in a variational autoencoder and provide a theoretical explanation to justify our method. In our framework, a connectionist temporal classification (CTC) -based force aligner acts as the approximate posterior, and text-to-duration works as the prior in the variational autoencoder. We evaluated our proposed method with a listening test and compared it with other TTS methods based on soft-attention or explicit duration modeling. The results showed that our systems rated between soft-attention-based methods (Transformer-TTS, Tacotron2) and explicit duration modeling-based methods (Fastspeech).

show abstract

Section: Methodsmentioning

confidence: 99%

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

Yasuda

Wang

Yamagishd

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…A semi-supervised approach utilizing both Mel spectrograms and prosodic features as inputs to a variational framework is proposed in [16]. In a similar approach to ours [17], aggregated continuous prosodic features (F0, mgc0, duration) are used for fine-grained prosody transfer. We differentiate our work by introducing discrete representations for arbitrary prosody control, as well as a method for disentanglement of phonetic and prosodic content.…”

Section: Related Workmentioning

confidence: 99%

Prosodic Clustering for Phoneme-Level Prosody Control in End-to-End Speech Synthesis

Vioni¹,

Christidou²,

Ellinas³

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

This paper presents a method for controlling the prosody at the phoneme level in an autoregressive attention-based text-to-speech system. Instead of learning latent prosodic features with a variational framework as is commonly done, we directly extract phoneme-level F0 and duration features from the speech data in the training set. Each prosodic feature is discretized using unsupervised clustering in order to produce a sequence of prosodic labels for each utterance. This sequence is used in parallel to the phoneme sequence in order to condition the decoder with the utilization of a prosodic encoder and a corresponding attention module. Experimental results show that the proposed method retains the high quality of generated speech, while allowing phoneme-level control of F0 and duration. By replacing the F0 cluster centroids with musical notes, the model can also provide control over the note and octave within the range of the speaker.

show abstract

“…Though sentence-level latent representation shows the ability to capture prosodic features from speech [9,16], it is short of fine-grained controllability and robustness in inter-speaker transfer. Recently, a train of researches have turned to fine-grained prosody modeling [17][18][19][20]. [17] introduced temporal structures on both speech and text sides for prosody embedding, which enabled pitch and amplitude manipulation at frame and phoneme levels.…”

Section: Fine-grained Prosody Modelingmentioning

confidence: 99%

“…[17] introduced temporal structures on both speech and text sides for prosody embedding, which enabled pitch and amplitude manipulation at frame and phoneme levels. [18] pre-computed prosody-related acoustic features of phonemes, and used them to reproduce a reference prosody on synthesized speech. [19] and [20] used fine-grained VAEs to learn local latent features under different model specifications.…”

Section: Fine-grained Prosody Modelingmentioning

confidence: 99%

Multi-Speaker Emotional Speech Synthesis with Fine-Grained Prosody Modeling

Chunhui

Wen

Liu

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We present an end-to-end system for multi-speaker emotional speech synthesis. In particular, our system learns emotion classes from just two speakers then generalizes these classes to other speakers from whom no emotional data was seen. We address the problem by integrating disentangled, fine-grained prosody features with global, sentence-level emotion embedding. These fine-grained features learn to represent local prosodic variations disentangled from speaker, tone and global emotion label. Compared to systems that model emotions at sentence level only, our method achieves higher ratings in naturalness and expressiveness, while retaining comparable speaker similarity ratings.

show abstract

Fine-Grained Robust Prosody Transfer for Single-Speaker Neural Text-To-Speech

Cited by 60 publications

References 0 publications

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

Prosodic Clustering for Phoneme-Level Prosody Control in End-to-End Speech Synthesis

Multi-Speaker Emotional Speech Synthesis with Fine-Grained Prosody Modeling

Contact Info

Product

Resources

About