YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone

Abstract

YourTTS brings the power of a multilingual approach to the task of zero-shot multi-speaker TTS. Our method builds upon the VITS model and adds several novel modifications for zeroshot multi-speaker and multilingual training. We achieved stateof-the-art (SOTA) results in zero-shot multi-speaker TTS and results comparable to SOTA in zero-shot voice conversion on the VCTK dataset. Additionally, our approach achieves promising results in a target language with a single-speaker dataset, opening possibilities for zero-shot multi-speaker TTS and zeroshot voice conversion systems in low-resource languages. Finally, it is possible to fine-tune the YourTTS model with less than 1 minute of speech and achieve state-of-the-art results in voice similarity and with reasonable quality. This is important to allow synthesis for speakers with a very different voice or recording characteristics from those seen during training.

A single speaker is almost all you need for automatic speech recognition

“…We used 3 languages/training datasets for the TTS model, as follows: English: VCTK [17] dataset, containing 44 hours of speech from 109 speakers, sampled at 48KHz. We divided the VCTK dataset into training, development and test subsets following [6]. To further increase the number of speakers for training, we used the subsets train-clean-100 and train-clean-360 from LibriTTS [18].…”

“…As the authors did not use a soundproof studio, the dataset contains some environmental noise. Following [6], we resampled the audios to 16Khz and used FullSubNet [22] as a denoiser. For development, we randomly selected 500 samples, leaving the rest for training.…”

“…Notably, most TTS systems are tailored for a single speaker, but many applications could benefit from the newspeaker synthesis, i.e., not seen during training, employing only a few seconds of the target speech. This approach is called zeroshot multi-speaker TTS (ZS-TTS) as in [4,5,6].…”

“…In our previous work [6], we presented YourTTS a zeroshot multi-speaker TTS model that showed good results for Portuguese using only a single-speaker dataset in the target language. In this paper, we combine the power of the YourTTS model with that of Wav2vec 2.0 [19] trained in a self-supervised way on 100 thousand hours of speech in 23 different languages [20].…”

Zero-Shot Long-Form Voice Cloning with Dynamic Convolution Attention

“…Recently, several non-autoregressive flow-based architectures for multispeaker TTS have been proposed [20,21]. These models can perform zero-shot voice cloning and potentially generalize to long utterances.…”