Mellotron: Multispeaker Expressive Voice Synthesis by Conditioning on Rhythm, Pitch and Global Style Tokens

Abstract

Mellotron is a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data. By explicitly conditioning on rhythm and continuous pitch contours from an audio signal or music score, Mellotron is able to generate speech in a variety of styles ranging from read speech to expressive speech, from slow drawls to rap and from monotonous voice to singing voice. Unlike other methods, we train Mellotron using only read speech data without alignments between text and audio. We evaluate our models using the LJSpeech and LibriTTS datasets. We provide F0 Frame Errors and synthesized samples that include style transfer from other speakers, singers and styles not seen during training, procedural manipulation of rhythm and pitch and choir synthesis.

Hierarchical Transfer Learning for Multilingual, Multi-Speaker, and Style Transfer DNN-Based TTS on Low-Resource Languages

“…Tacotron-GST [24] proposed modeling speech style using global style token (GST) by adding style token layer that consumes the reference encoder outputs [23] using a multihead attention scheme [43]. Recently, Mellotron [25] combined GST, pitch, and rhythm for style transferring and successfully reduced F0 frame error (FFE) significantly between synthesized audio and reference audio.…”

“…The encoder-decoder network can also be called spectogram prediction network that predicts spectrogram output from text input. The entire proposed multilingual multi-speaker TTS model, illustrated in Figure 1, is a sequence-to-sequence (seq-to-seq) Tacotron-2 network [13] with some additions: style embedding as in [24], pitch contour and attention map as in [25], language embedding, and speaker embedding. These additional networks are for handling multilingual, multi-speaker, and transfer of speaking style, pitch, and rhythm from a reference audio.…”

“…To carry out finer and detailed control, we add networks to condition melodic information such as pitch and rhythm. In addition to GST network, we adopt scheme in [25] to explicitly model expressive speech variables, such as fundamental frequency contour (F0) or pitch, and voicing decision (voiced/unvoiced), and rhythm variables. The pitch contour is extracted using the YIN algorithm [46] with a harmony threshold between 0.1 and 0.25 from the reference audio.…”

“…By learning the alignment map during training, we can control the rhythm during inference. Alignment map is extracted using a forcedaligner from <reference audio, transcription> pair data, as in [25]. TTS can produce the same rhythm as the reference audio using the extracted alignment map.…”

“…Besides producing high-quality synthetic sounds, DNN-based TTS introduces many possibilities to produce speech in various types of sounds, speech styles, and emotional states. E2E-prosody [23], Tacotron-GST [24], and Mellotron [25] proposed a DNN-based prosody model that has an important role in transferring a reference speaking style when generating synthesized speech. These works showed satisfactory results.…”