Learning emotions latent representation with CVAE for text-driven expressive audiovisual speech synthesis

Abstract

Great improvement has been made in the field of expressive audiovisual Textto-Speech synthesis (EAVTTS) thanks to deep learning techniques. However, generating realistic speech is still an open issue and researchers in this area have been focusing lately on controlling the speech variability. In this paper, we use different neural architectures to synthesize emotional speech. We study the application of unsupervised learning techniques for emotional speech modeling as well as methods for restructuring emotions representation to make it continuous and more flexible. This manipulation of the emotional representation should allow us to generate new styles of speech by mixing emotions. We first present our expressive audiovisual corpus. We validate the emotional content of this corpus with three perceptual experiments using acoustic only, visual only and audiovisual stimuli. After that, we analyze the performance of a fully connected neural network in learning characteristics specific to different emotions for the phone duration aspect and the acoustic and visual modalities. We also study the contribution of a joint and separate training of the acoustic and visual modalities in the quality of the generated synthetic speech. In the second part of this paper, we use a conditional variational auto-encoder (CVAE) architecture to learn a latent representation of emotions. We applied this method in an unsupervised manner to generate features of expressive speech. We used a

Guest editorial: Special issue on advances in deep learning based speech processing

“…The core idea is to use a non-autoregressive context decoder to generate acoustic features efficiently, and then add a shallow autoregressive acoustic decoder on top of the non-autoregressive context decoder to retrieve the temporal information of the acoustic signal. Dahmani et al [20] first presented an expressive audiovisual corpus, and then proposed to learn emotional latent representation with a conditional variational auto-encoder for text-driven expressive audiovisual speech synthesis. In Nallanthighal et al's study [21], they emphasized the importance of respiratory voice during TTS by exploring techniques for sensing breathing signal and breathing parameters from speech using deep learning architectures.…”