Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations

Abstract

This paper presents a method of sequence-tosequence (seq2seq) voice conversion using non-parallel training data. In this method, disentangled linguistic and speaker representations are extracted from acoustic features, and voice conversion is achieved by preserving the linguistic representations of source utterances while replacing the speaker representations with the target ones. Our model is built under the framework of encoder-decoder neural networks. A recognition encoder is designed to learn the disentangled linguistic representations with two strategies. First, phoneme transcriptions of training data are introduced to provide the references for leaning linguistic representations of audio signals. Second, an adversarial training strategy is employed to further wipe out speaker information from the linguistic representations. Meanwhile, speaker representations are extracted from audio signals by a speaker encoder. The model parameters are estimated by two-stage training, including a pretraining stage using a multi-speaker dataset and a fine-tuning stage using the dataset of a specific conversion pair. Since both the recognition encoder and the decoder for recovering acoustic features are seq2seq neural networks, there are no constrains of frame alignment and frame-by-frame conversion in our proposed method. Experimental results showed that our method obtained higher similarity and naturalness than the best non-parallel voice conversion method in Voice Conversion Challenge 2018. Besides, the performance of our proposed method was closed to the stateof-the-art parallel seq2seq voice conversion method.

StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks

“…Meanwhile, given the recent success of the sequence-tosequence (S2S) learning framework in various tasks, several VC methods based on S2S models have been proposed, including the ones we proposed previously [51]- [54]. While S2S models usually require parallel corpora for training, an attempt has also been made to train an S2S model using non-parallel utterances [55]. However, it requires phoneme transcriptions as auxiliary information for model training.…”

An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning

“…From this challenge, we observed that new speech waveform generation paradigms such as WaveNet and phone encoding have brought significant progress to the voice conversion field. Further improvements have been achieved in the follow up papers [295], [296] and new VC systems that exceed the challenge's best performance have already been reported.…”

Bootstrapping Non-Parallel Voice Conversion from Speaker-Adaptive Text-to-Speech

“…The proposed framework improves the stability of the generated speech but does not add any benefits in term of data efficiency, as parallel speech data of the source and target speakers is still a requirement. Recently, Jing-Xuan Zhang et al [18] have introduced a non-parallel sequence-tosequence voice conversion system that has a procedure that is similar to our: the initial model is trained with a TTS-like network to help disentangle linguistic representation and the model then adapted to the source and target speakers. However, the adaptation step of the framework proposed in [18] requires both source and target speaker speech as well as their transcripts, which increases the data demand for building a VC system with a particular voice.…”

“…For both parallel and non-parallel VC, the systems usually change the voice but are unable to change the duration of the utterance. Many recent researches focused on converting speaking rate along with voices by using sequence-to-sequence models [11,16,17,18], as speaking rate is also a speaker characteristic.…”