TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and Adversarial Training

Abstract

Non-parallel many-to-many voice conversion remains an interesting but challenging speech processing task. Recently, AutoVC, a conditional autoencoder based method, achieved excellent conversion results by disentangling the speaker identity and the speech content using informationconstraining bottlenecks. However, due to the pure autoencoder training method, it is difficult to evaluate the separation effect of content and speaker identity. In this paper, a novel voice conversion framework, named T ext Guided AutoVC(TGAVC), is proposed to more effectively separate content and timbre from speech, where an expected content embedding produced based on the text transcriptions is designed to guide the extraction of voice content. In addition, the adversarial training is applied to eliminate the speaker identity information in the estimated content embedding extracted from speech. Under the guidance of the expected content embedding and the adversarial training, the content encoder is trained to extract speaker-independent content embedding from speech. Experiments on AIShell-3 dataset show that the proposed model outperforms AutoVC in terms of naturalness and similarity of converted speech.

NIX-TTS: Lightweight and End-to-End Text-to-Speech Via Module-Wise Distillation

“…(Zhou et al, 2022b) is one of the firsts that introduce the ability to simulate emotion intensity and secondary emotion through a rank-based emotion attribute vector. (Tang et al, 2023) represents emotion as a vector embedding extracted from a pretrained speech emotion recognizer, which also allows the simulation of both characteristics by combining the hidden state of embedding.…”

“…As discussed in Section 2.3, there are mainly 2 candidates for our baselines, (Zhou et al, 2022b) and (Tang et al, 2023). We can only utilize (Zhou et al, 2022b) as our baseline as the latter is not open-sourced.…”

“…This would allow a wider representation of emotions, yet still underexplored in the emotional TTS design. To the best of our knowledge, it has only been adapted in a few neural emotional TTS, most notably (Zhou et al, 2022b;Tang et al, 2023), where they managed to simulate intensity-level and mixture of emotions through rank-based method and emotion embedding conditioning, respectively.…”

SUSing: SU-net for Singing Voice Synthesis

“…The method of HiFiSinger proposed by Chen et al [3], multi-scale adversarial training in both the acoustic model and vocoder was introduced to tackle the difficulty of singing modeling caused by the high sampling rate. A difference between singing voice synthesis and speech synthesis is that the prosody information in the song is more complex [9]- [12]. The vocal mechanism of singing and voice is different, and the pitch is relatively stable in singing.…”