VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion

Abstract

One-shot voice conversion (VC), which performs conversion across arbitrary speakers with only a single target-speaker utterance for reference, can be effectively achieved by speech representation disentanglement. Existing work generally ignores the correlation between different speech representations during training, which causes leakage of content information into the speaker representation and thus degrades VC performance. To alleviate this issue, we employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training, to achieve proper disentanglement of content, speaker and pitch representations, by reducing their inter-dependencies in an unsupervised manner. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations for retaining source linguistic content and intonation variations, while capturing target speaker characteristics. In doing so, the proposed approach achieves higher speech naturalness and speaker similarity than current state-of-the-art one-shot VC systems. Our code, pre-trained models and demo are available at https://github.com/Wendison/VQMIVC.

Parrot-Trained Adversarial Examples: Pushing the Practicality of Black-Box Audio Attacks against Speaker Recognition Models

“…These approaches can also be found in the audio domain, such as speech augmentation [63], [90], [64], [69], GAN-based speech synthesis [33], [65], [59], [21], and VAE-based speech synthesis [62], [55], [117]. Specifically, VC [94], [75], [70], [104], [31] is a specific data synthesis approach that can utilize a source speaker's speech to generate more voice samples that sound like a target speaker. Recent studies [107], [40] have revealed that it can be difficult for humans to distinguish whether the speech generated by a VC method is real or fake.…”

“…Existing VC studies [60], [61] have shown that intra-gender VC (e.g., female to female) appears to have better performance than inter-gender one (e.g., female to male). As a major difference between male and female voices is the pitch feature [70], [104], [75], which represents the basic frequency information of an audio signal, our intuition is that selecting a source speaker whose voice has the pitch feature similar to the target speaker may improve the VC performance. Therefore, for an attacker that knows a short speech sample of the target speaker to generate more parrot speech samples, the first step in our design is to find the best source speaker in a speech dataset (which can be a public dataset or the attacker's own dataset) such that the source speaker has the minimum average pitch distance to the target speaker.…”

Learning the Beauty in Songs: Neural Singing Voice Beautifier

“…Mainstream SVC models can be grouped into three categories : 1) parallel spectral feature mapping models, which learn the conversion function between source and target singers relying on parallel singing data (Villavicencio and Bonada, 2010;Kobayashi et al, 2015;Sisman et al, 2019); 2) Cycle-consistent Generative Adversarial Networks (CycleGAN) (Zhu et al, 2017;Kaneko et al, 2019), where an adversarial loss and a cycleconsistency loss are concurrently used to learn the forward and inverse mappings simultaneously (Sisman and Li, 2020); 3) encoder-decoder models, such as PPG-SVC (Li et al, 2021), which leverage a singing voice synthesis (SVS) system for SVC , and auto-encoder (Qian et al, 2019a;Wang et al, 2021b;Yuan et al, 2020) based SVC (Wang et al, 2021a). The models of the latter two categories can be utilized with nonparallel data.…”

Video-Guided Curriculum Learning for Spoken Video Grounding

“…Therefore, we set the model as depicted in Section 4 as our baseline model, which directly uses raw speech to localize the video fragments without any audio pretrain process, also we implement VSLNet [49] with audio by ourselves. Besides, we compare our proposed method with another audio pretrain model, VQCPC [43], which combine VQ-VAE [42] and CPC together to extract speech representations. We also use the DeepSpeech to translate the noise audio into text to test the performance of the baseline model with these ASR transcripts.…”