Analysing Deep Learning-Spectral Envelope Prediction Methods for Singing Synthesis

Bous, Frederik; Röebel, Axel

doi:10.23919/eusipco.2019.8903122

Cited by 3 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The auto-encoder is implemented as a fully convolutional neural network, where the encoder and decoder resemble mirrored structures. To benefit from the natural topology of the frequency axis, we treat the sequence of mel-spectrograms as images and perform 2D convolutions on them [47]. Through strides in frequency direction, the encoder reduces the frequency axis to length 1 but keeps the time axis unchanged.…”

Section: Network Architecturementioning

confidence: 99%

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

Bous

Röebel

2022

Information

Self Cite

View full text Add to dashboard Cite

In this publication, we present a deep learning-based method to transform the f0 in speech and singing voice recordings. f0 transformation is performed by training an auto-encoder on the voice signal’s mel-spectrogram and conditioning the auto-encoder on the f0. Inspired by AutoVC/F0, we apply an information bottleneck to it to disentangle the f0 from its latent code. The resulting model successfully applies the desired f0 to the input mel-spectrograms and adapts the speaker identity when necessary, e.g., if the requested f0 falls out of the range of the source speaker/singer. Using the mean f0 error in the transformed mel-spectrograms, we define a disentanglement measure and perform a study over the required bottleneck size. The study reveals that to remove the f0 from the auto-encoder’s latent code, the bottleneck size should be smaller than four for singing and smaller than nine for speech. Through a perceptive test, we compare the audio quality of the proposed auto-encoder to f0 transformations obtained with a classical vocoder. The perceptive test confirms that the audio quality is better for the auto-encoder than for the classical vocoder. Finally, a visual analysis of the latent code for the two-dimensional case is carried out. We observe that the auto-encoder encodes phonemes as repeated discontinuous temporal gestures within the latent code.

show abstract

Section: Network Architecturementioning

confidence: 99%

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

Bous

Röebel

2022

Information

Self Cite

View full text Add to dashboard Cite

show abstract

“…Non-Seq2Seq singing synthesizers include those based on autoregressive architectures [17,21,22], feed-forward CNN [23], and feed-forward GAN-based approaches [24,25].…”

Section: Relation To Prior Workmentioning

confidence: 99%

Sequence-to-Sequence Singing Synthesis Using the Feed-Forward Transformer

Blaauw

Bonada

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We propose a sequence-to-sequence singing synthesizer, which avoids the need for training data with pre-aligned phonetic and acoustic features. Rather than the more common approach of a content-based attention mechanism combined with an autoregressive decoder, we use a different mechanism suitable for feed-forward synthesis. Given that phonetic timings in singing are highly constrained by the musical score, we derive an approximate initial alignment with the help of a simple duration model. Then, using a decoder based on a feed-forward variant of the Transformer model, a series of self-attention and convolutional layers refines the result of the initial alignment to reach the target acoustic features. Advantages of this approach include faster inference and avoiding the exposure bias issues that affect autoregressive models trained by teacher forcing. We evaluate the effectiveness of this model compared to an autoregressive baseline, the importance of self-attention, and the importance of the accuracy of the duration model.

show abstract

Formant Analysis of Traditional Bulgarian Singing from Rhodope Region

Bozena

Zieliński

2019

2019 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA)

View full text Add to dashboard Cite

Analysing Deep Learning-Spectral Envelope Prediction Methods for Singing Synthesis

Cited by 3 publications

References 18 publications

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

Sequence-to-Sequence Singing Synthesis Using the Feed-Forward Transformer

Formant Analysis of Traditional Bulgarian Singing from Rhodope Region

Contact Info

Product

Resources

About