A Multi-Speaker Emotion Morphing Model Using Highway Networks and Maximum Likelihood Objective

Shankar, Ravi; Sager, Jacob; Venkataraman, Archana

doi:10.21437/interspeech.2019-2512

Cited by 16 publications

(9 citation statements)

References 17 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Studies have also revealed that the emotions can be expressed through universal principles that are shared across different individuals and cultures (Ekman, 1992;Manokara et al, 2021). This motivates the study of multispeaker (Shankar et al, 2019b(Shankar et al, , 2020, and speaker-independent emotional voice conversion (Zhou et al, 2020b;Choi and Hahn, 2021).…”

Section: Related Work Speech Emotion Conversionmentioning

confidence: 98%

Textless Speech Emotion Conversion using Discrete and Decomposed Representations

Kreuk¹,

Polyak²,

Copet³

et al. 2021

Preprint

View full text Add to dashboard Cite

Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion. First, we modify the speech content by translating the content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is superior to the baselines in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples and code will be publicly available under the following link: https://speechbot.github. io/emotion.

show abstract

Section: Related Work Speech Emotion Conversionmentioning

confidence: 98%

Textless Speech Emotion Conversion using Discrete and Decomposed Representations

Kreuk¹,

Polyak²,

Copet³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Moreover, there was also a study to combine hidden Markov model (HMM), GMM, and F0 segment selection [36] for spectrum and prosody conversion. Recent deep learning methods, such as deep belief network (DBN) [37], deep bidirectional long-short-term memory (DBLSTM) [38], highway neural network [10,39], sequence-to-sequence [40] and rule-based model [41] have achieved remarkable performance on emotion conversion. We note that the prior studies of emotion conversion do not provide an in-depth investigation of the disentanglement of emotional elements in speech, which will be the focus of this paper.…”

Section: Speaker a (Happy)mentioning

confidence: 99%

VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speech

Zhou¹,

Şişman²,

Li³

2020

Preprint

View full text Add to dashboard Cite

Emotional voice conversion (EVC) aims to convert the emotion of speech from one state to another while preserving the linguistic content and speaker identity. In this paper, we study the disentanglement and recomposition of emotional elements in speech through variational autoencoding Wasserstein generative adversarial network (VAW-GAN). We propose a speaker-dependent EVC framework based on VAW-GAN, that includes two VAW-GAN pipelines, one for spectrum conversion, and another for prosody conversion. We train a spectral encoder that disentangles emotion and prosody (F0) information from spectral features; we also train a prosodic encoder that disentangles emotion modulation of prosody (affective prosody) from linguistic prosody. At runtime, the decoder of spectral VAW-GAN is conditioned on the output of prosodic VAW-GAN. The vocoder takes the converted spectral and prosodic features to generate the target emotional speech. Experiments validate the effectiveness of our proposed method in both objective and subjective evaluations.

show abstract

“…In other words, they usually fail on multi-speaker datasets because features from different emotions tend to overlap considerably among various speakers. In an attempt to solve this, multi-speaker EVC systems [25], [26] that correspondingly modified the pitch and the energy, using a highway network [27] and a convolutional GAN network [28], were introduced. One of the recent studies [29] demonstrated variational autoencoding Wasserstein generative adversarial network (VAWGAN) based EVC utilizing the continuous wavelet transform (CWT) decomposition of F0 which allows the network to learn the speaker-independent emotion pattern across different speakers.…”

Section: Introductionmentioning

confidence: 99%

Sequence-to-Sequence Emotional Voice Conversion With Strength Control

Choi

Hahn

2021

IEEE Access

View full text Add to dashboard Cite

This paper proposes an improved emotional voice conversion (EVC) method with emotional strength and duration controllability. EVC methods without duration mapping generate emotional speech with identical duration to that of the neutral input speech. In reality, even the same sentences would have different speeds and rhythms depending on the emotions. To solve this, the proposed method adopts a sequence-to-sequence network with an attention module that enables the network to learn attention in the neutral input sequence should be focused on which part of the emotional output sequence. Besides, to capture the multi-attribute aspects of emotional variations, an emotion encoder is designed for transforming acoustic features into emotion embedding vectors. By aggregating the emotion embedding vectors for each emotion, a representative vector for the target emotion is obtained and weighted to reflect emotion strength. By introducing a speaker encoder, the proposed method can preserve speaker identity even after the emotion conversion. Objective and subjective evaluation results confirm that the proposed method is superior to other previous works. Especially, in emotion strength control, we achieve in getting successful results.

show abstract

A Multi-Speaker Emotion Morphing Model Using Highway Networks and Maximum Likelihood Objective

Cited by 16 publications

References 17 publications

Textless Speech Emotion Conversion using Discrete and Decomposed Representations

Textless Speech Emotion Conversion using Discrete and Decomposed Representations

VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speech

Sequence-to-Sequence Emotional Voice Conversion With Strength Control

Contact Info

Product

Resources

About