Principles for Learning Controllable TTS from Annotated and Latent Variation

Abstract

For building flexible and appealing high-quality speech synthesisers, it is desirable to be able to accommodate and reproduce fine variations in vocal expression present in natural speech. Synthesisers can enable control over such output properties by adding adjustable control parameters in parallel to their text input. If not annotated in training data, the values of these control inputs can be optimised jointly with the model parameters. We describe how this established method can be seen as approximate maximum likelihood and MAP inference in a latent variable model. This puts previous ideas of (learned) synthesiser inputs such as sentence-level control vectors on a more solid theoretical footing. We furthermore extend the method by restricting the latent variables to orthogonal subspaces via a sparse prior. This enables us to learn dimensions of variation present also within classes in coarsely annotated speech. As an example, we train an LSTM-based TTS system to learn nuances in emotional expression from a speech database annotated with seven different acted emotions. Listening tests show that our proposal successfully can synthesise speech with discernible differences in expression within each emotion, without compromising the recognisability of synthesised emotions compared to an identical system without learned nuances.

Learning emotions latent representation with CVAE for text-driven expressive audiovisual speech synthesis

“…Henter et al [39] and Zhu et al [34] succeeded in creating nuances of emotions without using emotion degree annotations, nevertheless, this work still relies on emotion labels as input.…”

Effective Emotion Transplantation in an End-to-End Text-to-Speech System

“…These are jointly trained with the weights of the TTS model using backpropagation [14]. For example, [10] and [11] trained embedding vectors in a supervised manner using emotion labels. Recently, several studies have adopted an unsupervised method in which embedding vectors are trained in a deep learning framework, but without annotated labels [12], [13], [15].…”

Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder

“…This paper addresses the problem of synthesizing expressive speech without relying on speech expression labels, which we refer to as unsupervised expressive speech synthesis (UESS). Many studies have reported that such labels are helpful for modeling complex audio data [5,6,4,7]. Unsupervised methods, however, are more desirable because expressive speech is easy to obtain from video hosting websites (e.g., Youtube) or audiobooks but annotating such sources is costly.…”