High-Quality Nonparallel Voice Conversion Based on Cycle-Consistent Adversarial Network

Fang, Fuming; Yamagishi, Junichi; Echizen, Isao; Lorenzo-Trueba, Jaime

doi:10.1109/icassp.2018.8462342

Cited by 108 publications

(74 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, the challenge in developing the non-parallel spectral conversion model has attracted many works within the recent years, such as: with the use of clustered spectral matching algorithms [12,13]; with adaptation/alignment of speaker model parameters [14,15]; with restricted Boltzmann machine [16]; with generative adversarial networks (GAN)-based methods [17,18]; and with variational autoencoder (VAE)-based frameworks [19,20,21,22]. In this work, we focus on the use of VAE-based system, due to its potential in employing latent space to represent common hidden aspects of speech signal, between different speakers, e.g., phonetical attributes.…”

Section: Introductionmentioning

confidence: 99%

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

In this paper, we present a novel technique for a non-parallel voice conversion (VC) with the use of cyclic variational autoencoder (CycleVAE)-based spectral modeling. In a variational autoencoder (VAE) framework, a latent space, usually with a Gaussian prior, is used to encode a set of input features. In a VAE-based VC, the encoded latent features are fed into a decoder, along with speaker-coding features, to generate estimated spectra with either the original speaker identity (reconstructed) or another speaker identity (converted). Due to the non-parallel modeling condition, the converted spectra can not be directly optimized, which heavily degrades the performance of a VAEbased VC. In this work, to overcome this problem, we propose to use CycleVAE-based spectral model that indirectly optimizes the conversion flow by recycling the converted features back into the system to obtain corresponding cyclic reconstructed spectra that can be directly optimized. The cyclic flow can be continued by using the cyclic reconstructed features as input for the next cycle. The experimental results demonstrate the effectiveness of the proposed CycleVAE-based VC, which yields higher accuracy of converted spectra, generates latent features with higher correlation degree, and significantly improves the quality and conversion accuracy of the converted speech.

show abstract

Section: Introductionmentioning

confidence: 99%

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…As the sample rate of the codebook embeddings of our system was 320 times smaller than the speech samples, the Wavenet couldn't produced satisfying result. GANs are known to be effective for achieving high-quality voice conversion with clean input data [29,30]. However, our task is more challenging due to the fact that our generated voice will always have some distortion.…”

Section: Resultsmentioning

confidence: 99%

VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019

et al. 2019

View full text Add to dashboard Cite

We describe our submitted system for the ZeroSpeech Challenge 2019. The current challenge theme addresses the difficulty of constructing a speech synthesizer without any text or phonetic labels and requires a system that can (1) discover subword units in an unsupervised way, and (2) synthesize the speech with a target speaker's voice. Moreover, the system should also balance the discrimination score ABX, the bit-rate compression rate, and the naturalness and the intelligibility of the constructed voice. To tackle these problems and achieve the best tradeoff, we utilize a vector quantized variational autoencoder (VQ-VAE) and a multi-scale codebook-tospectrogram (Code2Spec) inverter trained by mean square error and adversarial loss. The VQ-VAE extracts the speech to a latent space, forces itself to map it into the nearest codebook and produces compressed representation. Next, the inverter generates a magnitude spectrogram to the target voice, given the codebook vectors from VQ-VAE. In our experiments, we also investigated several other clustering algorithms, including K-Means and GMM, and compared them with the VQ-VAE result on ABX scores and bit rates. Our proposed approach significantly improved the intelligibility (in CER), the MOS, and discrimination ABX scores compared to the official ZeroSpeech 2019 baseline or even the topline.

show abstract

“…5 is the full model). We also compared CycleGAN-VC2 with two stateof-the-art methods: CycleGAN-VC [29] and frame-based CycleGAN [30] (our reimplementation; we additionally used L id for stabilizing training). The comparison of one-step and two-step adversarial losses (nos.…”

Section: Objective Evaluationmentioning

confidence: 99%

Cyclegan-VC2: Improved Cyclegan-based Non-parallel Voice Conversion

Kaneko

Kameoka

Tanaka

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

197

140

View full text Add to dashboard Cite

Non-parallel voice conversion (VC) is a technique for learning the mapping from source to target speech without relying on parallel data. This is an important task, but it has been challenging due to the disadvantages of the training conditions. Recently, CycleGAN-VC has provided a breakthrough and performed comparably to a parallel VC method without relying on any extra data, modules, or time alignment procedures. However, there is still a large gap between the real target and converted speech, and bridging this gap remains a challenge. To reduce this gap, we propose CycleGAN-VC2, which is an improved version of CycleGAN-VC incorporating three new techniques: an improved objective (twostep adversarial losses), improved generator (2-1-2D CNN), and improved discriminator (PatchGAN). We evaluated our method on a non-parallel VC task and analyzed the effect of each technique in detail. An objective evaluation showed that these techniques help bring the converted feature sequence closer to the target in terms of both global and local structures, which we assess by using Mel-cepstral distortion and modulation spectra distance, respectively. A subjective evaluation showed that CycleGAN-VC2 outperforms CycleGAN-VC in terms of naturalness and similarity for every speaker pair, including intra-gender and inter-gender pairs. 1

show abstract

High-Quality Nonparallel Voice Conversion Based on Cycle-Consistent Adversarial Network

Cited by 108 publications

References 27 publications

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019

Cyclegan-VC2: Improved Cyclegan-based Non-parallel Voice Conversion

Contact Info

Product

Resources

About