Improve Cross-Lingual Text-To-Speech Synthesis on Monolingual Corpora with Pitch Contour Information

Zhan, Haoyue; Zhang, Haitong; Ou, Wenjie; Lin, Yue

doi:10.21437/interspeech.2021-474

Cited by 5 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The attention module is shared by both streams to keep inter-stream synchrony. Following previous work [13], location sensitive attention mechanism [17] is adopted here to align text encoder outputs to acoustic feature sequences. The predicted mel-cepstra, logF0 and energy at the previous frame are passed to the prenet.…”

Section: Model Architecturementioning

confidence: 99%

“…Following the previous work [13], an adversarial speaker classifier [18] with a gradient reversal layer is applied to the concatenated encoder output. It follows the principle of domain adversarial training [19] to remove the residual speaker information in the encoder output.…”

Section: Model Architecturementioning

confidence: 99%

“…For example, adding tone embeddings was proposed to improve the naturalness of synthesizing tones in Chinese [12]. Predicting a binary fundamental frequency profile for each phoneme was employed to enhance Japanese synthesis performance in a multilingual model [13]. Considering that some languages may have similar phoneme sets, while others may have similar prosodic properties, it is reasonable to share the pronunciation knowledge and the prosody knowledge among languages separately in a unified multilingual TTS model.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Decoupled Pronunciation and Prosody Modeling in Meta-Learning-based Multilingual Speech Synthesis

Peng¹,

Ling²

2022

Interspeech 2022

View full text Add to dashboard Cite

This paper presents a method of decoupled pronunciation and prosody modeling to improve the performance of meta-learning-based multilingual speech synthesis.The baseline meta-learning synthesis method adopts a single text encoder with a parameter generator conditioned on language embeddings and a single decoder to predict mel-spectrograms for all languages. In contrast, our proposed method designs a two-stream model structure that contains two encoders and two decoders for pronunciation and prosody modeling, respectively, considering that the pronunciation knowledge and the prosody knowledge should be shared in different ways among languages. In our experiments, our proposed method effectively improved the intelligibility and naturalness of multilingual speech synthesis comparing with the baseline meta-learning synthesis method.

show abstract

Section: Model Architecturementioning

confidence: 99%

Section: Model Architecturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Decoupled Pronunciation and Prosody Modeling in Meta-Learning-based Multilingual Speech Synthesis

Peng¹,

Ling²

2022

Interspeech 2022

View full text Add to dashboard Cite

show abstract

“…Based on the disentanglement strategy, the existing crosslingual approaches can be roughly divided into implicitbased and explicit-based methods [9]. Implicit-based methods mainly study the unified linguistic/phonetic representations across languages to disentangle language and speaker timbre implicitly [11], [12], [13], [14], [15], [16]. On the other hand, to further solve the foreign accent problem, the explicit-based methods prefer to adopt adversarial learning [1], [7], [9], [17] or mutual information [6] to minimize the correlation between different speech factors, thus encouraging the model to automatically learn disentangled linguistic representations.…”

Section: Introductionmentioning

confidence: 99%

DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech — A Study Between English and Mandarin

Li,

Hu,

Cong

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

While the performance of cross-lingual TTS based on monolingual corpora has been significantly improved recently, generating cross-lingual speech still suffers from the foreign accent problem, leading to limited naturalness. Besides, current cross-lingual methods ignore modeling emotion, which is indispensable paralinguistic information in speech delivery. In this paper, we propose DiCLET-TTS, a Diffusion model based Cross-Lingual Emotion Transfer method that can transfer emotion from a source speaker to the intra-and cross-lingual target speakers. Specifically, to relieve the foreign accent problem while improving the emotion expressiveness, the terminal distribution of the forward diffusion process is parameterized into a speaker-irrelevant but emotion-related linguistic prior by a prior text encoder with the emotion embedding as a condition. To address the weaker emotional expressiveness problem caused by speaker disentanglement in emotion embedding, a novel orthogonal projection based emotion disentangling module (OP-EDM) is proposed to learn the speaker-irrelevant but emotiondiscriminative embedding. Moreover, a condition-enhanced DPM decoder is introduced to strengthen the modeling ability of the speaker and the emotion in the reverse diffusion process to further improve emotion expressiveness in speech delivery. Cross-lingual emotion transfer experiments show the superiority of DiCLET-TTS over various competitive models and the good design of OP-EDM in learning speaker-irrelevant but emotiondiscriminative embedding.

show abstract

“…[6,7] and [8,9] use Unicode bytes and Phonetic Posterior-Grams (PPGs) respectively as common phonetic set to build cross-lingual TTS systems and all get improvements compared to their baselines. Moreover, [10,11] convert all the graphemes of different languages into a same International Phonetic Alphabet (IPA) set to facilitate cross-lingual modeling and the experiments in [11] show the privilege of IPA over language-dependent phonemes.…”

Section: Introductionmentioning

confidence: 99%

Improving Cross-lingual Speech Synthesis with Triplet Training Scheme

Ye¹,

Zhou²,

Su³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent advances in cross-lingual text-to-speech (TTS) made it possible to synthesize speech in a language foreign to a monolingual speaker. However, there is still a large gap between the pronunciation of generated cross-lingual speech and that of native speakers in terms of naturalness and intelligibility. In this paper, a triplet training scheme is proposed to enhance the cross-lingual pronunciation by allowing previously unseen content and speaker combinations to be seen during training. Proposed method introduces an extra fine-tune stage with triplet loss during training, which efficiently draws the pronunciation of the synthesized foreign speech closer to those from the native anchor speaker, while preserving the non-native speaker's timbre. Experiments are conducted based on a state-of-the-art baseline cross-lingual TTS system and its enhanced variants. All the objective and subjective evaluations show the proposed method brings significant improvement in both intelligibility and naturalness of the synthesized cross-lingual speech.

show abstract

Improve Cross-Lingual Text-To-Speech Synthesis on Monolingual Corpora with Pitch Contour Information

Cited by 5 publications

References 19 publications

Decoupled Pronunciation and Prosody Modeling in Meta-Learning-based Multilingual Speech Synthesis

Decoupled Pronunciation and Prosody Modeling in Meta-Learning-based Multilingual Speech Synthesis

DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech — A Study Between English and Mandarin

Improving Cross-lingual Speech Synthesis with Triplet Training Scheme

Contact Info

Product

Resources

About