Non-Parallel Emotion Conversion Using a Deep-Generative Hybrid Network and an Adversarial Pair Discriminator

Abstract

We introduce a novel method for emotion conversion in speech that does not require parallel training data. Our approach loosely relies on a cycle-GAN schema to minimize the reconstruction error from converting back and forth between emotion pairs. However, unlike the conventional cycle-GAN, our discriminator classifies whether a pair of input real and generated samples corresponds to the desired emotion conversion (e.g., A → B) or to its inverse (B → A). We will show that this setup, which we refer to as a variational cycle-GAN (VC-GAN), is equivalent to minimizing the empirical KL divergence between the source features and their cyclic counterpart. In addition, our generator combines a trainable deep network with a fixed generative block to implement a smooth and invertible transformation on the input features, in our case, the fundamental frequency (F0) contour. This hybrid architecture regularizes our adversarial training procedure. We use crowd sourcing to evaluate both the emotional saliency and the quality of synthesized speech. Finally, we show that our model generalizes to new speakers by modifying speech produced by Wavenet.

Learning to Model Prosodic and Spectral Features for Non-parallel Emotive Speech Conversion

“…Thus, in this work we modify both the F 0 contour, associated with the excitation source characteristics, along with the spectral envelope, associated with the vocal tract system characteristics. Recently [30,31], convolutional network based cycleGANs were considered for modeling the F 0 values. In [30], a convolutional network based cycleGAN was considered for modeling the F 0 values represented using a continuous wavelet transform (CWT) for emotion conversion.…”

“…In [30], a convolutional network based cycleGAN was considered for modeling the F 0 values represented using a continuous wavelet transform (CWT) for emotion conversion. In [31], the generator network consisted of a convolutional neural network followed by a deterministic block with static parameters for modeling the F0 contour. We introduce auto-regressive generator networks to better model the multi-resolution temporal coherence of the F 0 contour.…”

Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset

“…There have been studies on deep learning approaches for emotional voice conversion that do not require parallel training data, such as cycle-consistent adversarial network (CycleGAN)-based [17,18] and autoencoder-based frameworks [19,20]. However, they are typically designed for a fixed set of conversion pairs.…”

Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-Stage Sequence-to-Sequence Training

“…Inspired by the success in speaker voice conversion, these methods are adopted to model both spectral and prosodic parameters for emotional voice conversion. Successful attempts include GMM [11], sparse representation [12], deep bi-directional long-short-term memory (BLSTM) network [13], GAN-based [14][15][16][17] and autoencoder-based [18][19][20][21] methods. These frameworks model the mapping on a frame-by-frame basis.…”