Emotional Audio-Visual Speech Synthesis Based on PAD

Abstract

Recent research in zero-shot speech synthesis has made significant progress in speaker similarity. However, current efforts focus on timbre generalization rather than prosody modeling, which results in limited naturalness and expressiveness. To address this, we introduce a novel speech synthesis model trained on large-scale datasets, including both timbre and hierarchical prosody modeling. As timbre is a global attribute closely linked to expressiveness, we adopt a global vector to model speaker timbre while guiding prosody modeling. Besides, given that prosody contains both global consistency and local variations, we introduce a diffusion model as the pitch predictor and employ a prosody adaptor to model prosody hierarchically, further enhancing the prosody quality of the synthesized speech. Experimental results show that our model not only maintains comparable timbre quality to the baseline but also exhibits better naturalness and expressiveness. The synthesized samples can be found at: https://rxy-j.github.io/HPMD-TTS/

Continuous Estimation of Emotions in Speech by Dynamic Cooperative Speaker Models

“…Results show that even though the SVR provides the best performance in the validation of each SSRM for both arousal and valence, the PLS algorithm is more robust to overfitting and thus produces significantly improved performance. Our conclusion is that weak predictors are indeed more suitable to perform boosting than more sophisticated algorithms [50].…”

“…Results confirm that the prediction of arousal from acoustic features provides significantly better results than for valence. The combination of weak predictors (PLS) in the CRM, which is similar to a boosting strategy [50], provides a performance that is comparable with the one obtained with more complex machine learning methods that are trained on a full set of speakers [26], [37].…”

Extracting News Information Based on Webpage Segmentation and Parsing DOM Tree Reversely

“…However, the method is presented in accordance with the project about the analysis of the net-mediated public sentiment, so the universality of the method is insufficient, and it is not recommended to apply the method to extract information from non-news webpages. We will set out to extract news videos and pictures from news webpages [6] in the future work.…”

Head and facial gestures synthesis using PAD model for an expressive talking avatar

“…The PAD model essentially shades the modeling of head and facial gestures from the highlevel text semantics, so that we can focus on mapping the PAD descriptors to visual motion features. Toward the PAD parameterization for input text, we adopt the heuristics that are proposed in the PAD based expressive text-to-speech synthesis [24,38]. To extend our approach to talking avatar in other languages, similar PAD parameterizations need to be devised according to the specific language.…”