Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis

Abstract

Generating versatile and appropriate synthetic speech requires control over the output expression separate from the spoken text. Important non-textual speech variation is seldom annotated, in which case output control must be learned in an unsupervised fashion. In this paper, we perform an indepth study of methods for unsupervised learning of control in statistical speech synthesis. For example, we show that popular unsupervised training heuristics can be interpreted as variational inference in certain autoencoder models. We additionally connect these models to VQ-VAEs, another, recently-proposed class of deep variational autoencoders, which we show can be derived from a very similar mathematical argument. The implications of these new probabilistic interpretations are discussed. We illustrate the utility of the various approaches with an application to acoustic modelling for emotional speech synthesis, where the unsupervised methods for learning expression control (without access to emotional labels) are found to give results that in many aspects match or surpass the previous best supervised approach.

Audio representations for deep learning in sound synthesis: A review

“…Other implementations have included cepstral coefficients [4] [36] or a variety of linguistic and acoustic features [37] [13]. Finally, widely recommended parameters have also been extracted by the WORLD vocoder [6] [38] [39].…”

“…Following these techniques, the generated sound can be conditioned on specific traits such as speaker's voice [47] [27], independent pitch [3] [48] [36], linguistic features [49] [17] or latent representations [4] [45]. Instead of one-hot-embeddings, some implementations have also used a confusion matrix to capture a variation of emotions [39], while others provided supplementary positional information of each segment conditioning music to the artist or genre [43]. After training, the user is able to decide between the conditioning properties of the synthesised sound.…”

Analysis and Assessment of Controllability of an Expressive Deep Learning-based TTS system

“…Concerning controllable speech synthesis, [1] proposed to use a VAE and deploy a speech synthesis system that combines VAE with VoiceLoop [15]. Some other researches have used the concept of VAE [4,5] for controllable speech synthesis. In [5], the authors combine VAE and GMM and call it GMVAE.…”

“…In [5], the authors combine VAE and GMM and call it GMVAE. For more details concerning the different variants of such methods, an in-depth study of methods for unsupervised learning of control in speech synthesis is given in [4]. These works show that it is possible to build a latent space leading to variables that can be used to control the style of synthesized speech.…”

MASS: Multi-task Anthropomorphic Speech Synthesis Framework

“…There are several emotional speech synthesis methods. For example, emotional speech synthesis methods based on reference audio feature embedding [15,18], variational auto-encoder (VAE) and normalizing flows [1], etc. Furthermore, Lee et al [37] proposed emotional end-to-end neural speech synthesizer based on Tacotron.…”