A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural Songs

Abstract

Abstract:We recently presented a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Nonetheless, compared to modeling waveform directly, ways of effectively handling higher-dimensional outputs, multiple feature streams and regularization become more important with our approach. In this work, we extend our proposed system to include additional components for predicting F0 and phonetic timings from a musical score with lyrics. These expression-related features are learned together with timbrical features from a single set of natural songs. We compare our method to existing statistical parametric, concatenative, and neural network-based approaches using quantitative metrics as well as listening tests.

Sequence-to-Sequence Singing Synthesis Using the Feed-Forward Transformer

“…Our proposed system uses 64-dimensional input features similar to [17], extracted with a 10 ms hop time. A reduction factor, r = 2, is used.…”

“…Non-Seq2Seq singing synthesizers include those based on autoregressive architectures [17,21,22], feed-forward CNN [23], and feed-forward GAN-based approaches [24,25].…”

A Vocoder Based Method for Singing Voice Extraction

“…The system decomposes a speech signal into the fundamental frequency f 0, harmonic spectral envelope and aperiodicity envelope. It has been proved that these parameters can be used to reconstruct a high quality synthesis of speech signals, even after dimensionality reduction techniques have been applied to the parameters [15].…”

Special Issue on “Sound and Music Computing”

“…It is applied to a specific piano, and the results outperform the earlier methods in note-level polyphonic piano music transcription. Blaauw and Bonada [7] describe a singing synthesizer based on deep neural networks called the Neural Parametric Singing Synthesizer (NPSS), which can generate high-quality singing when a musical score and lyrics are given as the input. The NPSS can learn the timbre and expressive features of a singer from a small set of recordings.…”