Speech Synthesis With Mixed Emotions

Abstract

Emotional speech synthesis aims to synthesize human voices with various emotional effects. The current studies are mostly focused on imitating an averaged style belonging to a specific emotion type. In this paper, we seek to generate speech with a mixture of emotions at run-time. We propose a novel formulation that measures the relative difference between the speech samples of different emotions. We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework. During the training, the framework does not only explicitly characterize emotion styles but also explores the ordinal nature of emotions by quantifying the differences with other emotions. At run-time, we control the model to produce the desired emotion mixture by manually defining an emotion attribute vector. The objective and subjective evaluations have validated the effectiveness of the proposed framework. To our best knowledge, this research is the first study on modelling, synthesizing and evaluating mixed emotions in speech.

NIX-TTS: Lightweight and End-to-End Text-to-Speech Via Module-Wise Distillation

“…This would allow a wider representation of emotions, yet still underexplored in the emotional TTS design. To the best of our knowledge, it has only been adapted in a few neural emotional TTS, most notably (Zhou et al, 2022b;Tang et al, 2023), where they managed to simulate intensity-level and mixture of emotions through rank-based method and emotion embedding conditioning, respectively.…”

End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks

“…One way in which emotions are expressed by individuals in social interactions is via speech signals [1]. In the context of human-machine interaction systems, the generation of spoken dialogue is a fundamental facet of natural interaction between humans and machines [2,3]. More importantly, to improve the naturalness of machine communication, the generation of emotionally expressive speech is required.…”

“…To overcome this, the circumplex model [11] captures emotional expressions using two continuous and independent dimensions, i.e., arousal (relaxed or passive vs. aroused or activated) and valence (positive vs. negative) [12,13]. In SEC research, efforts have also been made to control the intensity of categorical emotion representations, e.g., using mixed emotion representations [3] or modeling emotion intensity as an auxiliary task [14]. Note that SEC using the dimensional representations directly archives intensity control, as opposed to an additional effort in the categorical representation case.…”

“…During inference, the latent representation encoding emotion is modified to achieve emotion conversion. Existing works have primarily employed variational auto-encoders (VAE) [15,19] and sequence-to-sequence encoder-decoders [3,14] to achieve disentanglement.…”

Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration

“…However, the existing methods in adaptive TTS face significant limitations when applied to emotional speech generation, whose objective is to synthesize the speech with the desired emotion, in zero-shot scenarios. Specifically, existing approaches for zero-shot adaptive TTS do not consider the emotion-controllable generation, while the most of existing approaches for emotional TTS [9,10,11,12,13,14] do not take zero-shot scenarios into account. These limitations necessitate the development of new methods that can effectively address both requirements in TTS systems, as depicted in Figure 1.…”