Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations

Chou, Ju-Chieh; Yeh, Cheng-chieh; Lee, Hung-yi; Lee, Lin-Shan

doi:10.48550/arxiv.1804.02812

Cited by 20 publications

(41 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, Hung et al [19] explored disentangled representations for timbre and pitch of musical sounds useful for music editing. Chou et al [20] explored the disentanglement of speaker characteristics from linguistic content in speech signals for voice conversion. Chen et al [21] explored the idea of disentangling phonetic and speaker information for the task of audio representation.…”

Section: Representation Learningmentioning

confidence: 99%

CDPAM: Contrastive Learning for Perceptual Audio Similarity

Manocha

Jin

Zhang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Many speech processing methods based on deep learning require an automatic and differentiable audio metric for the loss function. The DPAM approach of Manocha et al.[1] learns a full-reference metric trained directly on human judgments, and thus correlates well with human perception. However, it requires a large number of human annotations and does not generalize well outside the range of perturbations on which it was trained. This paper introduces CDPAM -a metric that builds on and advances DPAM. The primary improvement is to combine contrastive learning and multi-dimensional representations to build robust models from limited data. In addition, we collect human judgments on triplet comparisons to improve generalization to a broader range of audio perturbations. CDPAM correlates well with human responses across nine varied datasets. We also show that adding this metric to existing speech synthesis and enhancement methods yields significant improvement, as measured by objective and subjective tests.

show abstract

Section: Representation Learningmentioning

confidence: 99%

CDPAM: Contrastive Learning for Perceptual Audio Similarity

Manocha

Jin

Zhang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Vector quantization based methods [14] are further proposed to model content information as discrete distributions which are more related to the distribution of phonetic information. An auxiliary adversarial speaker classifier is adopted [15] to encourage the encoder to cast away speaker information from content information by minimizing the mutual information between their representations [16].…”

Section: Introductionmentioning

confidence: 99%

Adversarially learning disentangled speech representations for robust multi-factor voice conversion

Wang¹,

Li²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Factorizing speech as disentangled speech representations is vital to achieve highly controllable style transfer in voice conversion (VC). Conventional speech representation learning methods in VC only factorize speech as speaker and content, lacking controllability on other prosody-related factors. State-of-the-art speech representation learning methods for more speech factors are using primary disentangle algorithms such as random resampling and ad-hoc bottleneck layer size adjustment, which however is hard to ensure robust speech representation disentanglement. To increase the robustness of highly controllable style transfer on multiple factors in VC, we propose a disentangled speech representation learning framework based on adversarial learning. Four speech representations characterizing content, timbre, rhythm and pitch are extracted, and further disentangled by an adversarial network inspired by BERT. The adversarial network is used to minimize the correlations between the speech representations, by randomly masking and predicting one of the representations from the others. A word prediction network is also adopted to learn a more informative content representation. Experimental results show that the proposed speech representation learning framework significantly improves the robustness of VC on multiple factors by increasing conversion rate from 48.2% to 57.1% and ABX preference exceeding by 31.2% compared with state-of-the-art method.

show abstract

“…While disentangling natural image representation has been studied extensively, much less work has focused on natural speech, leaving a rather large void in the understanding of this problem. In this paper, we first present a short review and comparison of two representative efforts on this topic [6,7], where both efforts involve using an auto-encoder and can be applied to the same task (i.e., voice conversion), but the key disentangling algorithms and underlying ideas are very different.…”

Section: Introductionmentioning

confidence: 99%

“…Different from [6], in [7], the authors propose a supervised approach based on adversarial training [11,12,13,14](illustrated in Figure 2 (left)). In addition to a regular autoencoder, the authors add a regularization term in its objective function to force the latent variables (i.e., the encoding) to not contain speaker information.…”

Section: Introductionmentioning

confidence: 99%

“…That is, in [6], z1 and z2 are in fact corresponding to general fast-changing and slow-changing in-formation, i.e., z1 may contain other fast-changing information such as emotion, while z2 may contain slow-changing factors such as background and channel noise. In [7], the authors actually separate out speaker information and general non-speaker information, which may contain a lot of detailed information. Coarse-grained disentangled representations are enough for some tasks, such as voice speaker conversion, but might be limited for other tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation