Learning Latent Representations for Speech Generation and Transformation

Hsu, Wei-Ning; Yu, Zhongbo; Glass, James

doi:10.21437/interspeech.2017-349

Cited by 99 publications

(76 citation statements)

References 17 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It can be clearly seen that the proposed CycleVAE-based VC generates latent features with higher correlation degree compared to conventional VAE. As studied in [32], higher cosine similarities would be produced by latent attributes that represent either equal phonetic space or equal speaker identities. Hence, Cycle-VAE is more likely to give latent representations that are closer to phonetic domain due to different speaker identities.…”

Section: Objective Evaluationmentioning

confidence: 99%

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

In this paper, we present a novel technique for a non-parallel voice conversion (VC) with the use of cyclic variational autoencoder (CycleVAE)-based spectral modeling. In a variational autoencoder (VAE) framework, a latent space, usually with a Gaussian prior, is used to encode a set of input features. In a VAE-based VC, the encoded latent features are fed into a decoder, along with speaker-coding features, to generate estimated spectra with either the original speaker identity (reconstructed) or another speaker identity (converted). Due to the non-parallel modeling condition, the converted spectra can not be directly optimized, which heavily degrades the performance of a VAEbased VC. In this work, to overcome this problem, we propose to use CycleVAE-based spectral model that indirectly optimizes the conversion flow by recycling the converted features back into the system to obtain corresponding cyclic reconstructed spectra that can be directly optimized. The cyclic flow can be continued by using the cyclic reconstructed features as input for the next cycle. The experimental results demonstrate the effectiveness of the proposed CycleVAE-based VC, which yields higher accuracy of converted spectra, generates latent features with higher correlation degree, and significantly improves the quality and conversion accuracy of the converted speech.

show abstract

Section: Objective Evaluationmentioning

confidence: 99%

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…In the third layer, we also stride along the frequency axis, but perform this only once to not lose too much frequency information. The fifth convolutional layer has a filter of size 48 in the frequency domain, which captures frequency relationships in a larger range of the CQT, as shown to be successful in [17] and [22]. The error function is the Mean-Squared Error (MSE) between the pitch shift estimate and ground truth over the full sequence of notes in a performance.…”

Section: Neural Network Structurementioning

confidence: 99%

Deep Autotuner: A Pitch Correcting Network for Singing Performances

Wager

Tzanetakis

Wang³

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We introduce a data-driven approach to automatic pitch correction of solo singing performances. The proposed approach predicts notewise pitch shifts from the relationship between the respective spectrograms of the singing and accompaniment. This approach differs from commercial systems, where vocal track notes are usually shifted to be centered around pitches in a user-defined score, or mapped to the closest pitch among the twelve equal-tempered scale degrees. The proposed system treats pitch as a continuous value rather than relying on a set of discretized notes found in musical scores, thus allowing for improvisation and harmonization in the singing performance. We train our neural network model using a dataset of 4,702 amateur karaoke performances selected for good intonation. Our model is trained on both incorrect intonation, for which it learns a correction, and intentional pitch variation, which it learns to preserve. The proposed deep neural network with gated recurrent units on top of convolutional layers shows promising performance on the real-world score-free singing pitch correction taskautotuning.

show abstract

“…This work makes a comparison of different models, including VAEs, However, this work works with raw audio instead of MIDI files since its objective is to model sound and not music generation. VAEs have been used also for processing of speech signals with the objective of making modifications in some attributes of the speakers [16] [17] 3 Variational Autoencoders Variational Autoencoders arose from the evolution of autoencoders [18] [5] [19]. Both techniques aim to codify a set of data into a smaller vector, and then reconstructing the original data from this vector.…”

Section: State Of the Artmentioning

confidence: 99%

Classical Music Prediction and Composition by Means of Variational Autoencoders

et al. 2020

View full text Add to dashboard Cite

This paper proposes a new model for music prediction based on Variational Autoencoders (VAEs). In this work, VAEs are used in a novel way in order to address two different problems: music representation into the latent space, and using this representation to make predictions of the future values of the musical piece. This approach was trained with different songs of a classical composer. As a result, the system can represent the music in the latent space, and make accurate predictions. Therefore, the system can be used to compose new music either from an existing piece or from a random starting point. An additional feature of this system is that a small dataset was used for training. However, results show that the system is able to return accurate representations and predictions in unseen data.

show abstract

Learning Latent Representations for Speech Generation and Transformation

Cited by 99 publications

References 17 publications

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Deep Autotuner: A Pitch Correcting Network for Singing Performances

Classical Music Prediction and Composition by Means of Variational Autoencoders

Contact Info

Product

Resources

About