Litesing: Towards Fast, Lightweight and Expressive Singing Voice Synthesis

Abstract

LiteSing proposed in this paper is a high-quality singing voice synthesis (SVS) system, which is fast, lightweight and expressive. This model mainly stacks several non-autoregressive WaveNet blocks in the encoder and decoder under a generative adversarial architecture, predicts full conditions from the musical score, and generates acoustic features from these conditions. The full conditions in this paper consist of dynamic spectrogram energy, voiced/unvoiced (V/UV) decision and dynamic pitch curve, which are proven related to the expressiveness. We predict the pitch and the timbre features separately, avoiding the interdependence between these two features. Instead of neural network vocoders, a parametric WORLD vocoder is employed for the pitch curve consistency. Experiment results show that LiteSing outperforms the baseline model using feed-forward Transformer by 1.386 times faster on inference speed, 15 times smaller on training parameters number, and achieves a similar MOS on sound quality. Through an A/B test, LiteSing achieves 67.3% preference rate over baseline in pitch curve and dynamic spectrogram energy prediction. which demonstrates the advantage of LiteSing over the other compared models.

Expressive Singing Synthesis Using Local Style Token and Dual-path Pitch Encoder

“…To qualitatively examine the controllability of the proposed methods, we tried various style modifications by manipulating the initial LST sequence and f0 contour 2 .…”

“…Recently, interest in research on the SVS system that can reflect musical expression is increasing. A method of explicitly modeling information such as pitch curves, energy, V/UV., which can be extracted directly from the vocal signal, was proposed in [2]. [3] proposed a method to interpret the music score more naturally by introducing a module that predicts the difference between the actual singing and the score.…”

“…Singing voice synthesis (SVS) is the task of generating a natural singing voice from a given musical score. With the development of various deep generative models, research on synthesizing high-quality singing voice has been emerging recently [1,2,3,4]. As the performance of the SVS improves, there are increasing cases in which the technology is applied to the production of actual music content [5].…”

Singing Voice Synthesis with Vibrato Modeling and Latent Energy Representation

“…Singing voice synthesis (SVS) systems [1]- [7] generate singing voices from musical scores which contain music information such as lyrics, tempo, pitch, etc. SVS is similar to the text-to-speech (TTS) task [8]- [13] in terms of generating speech from text.…”

“…For example, to predict pitch feature better, Yi et al [18] utilized deep autoregressive network to capture the dependencies among the consecutive acoustic features. Zhuang et al [1] separated the pitch feature from the acoustic feature to avoid the interdependence between these pitch features and the timbre features. Ren et al [9] introduced the pitch and energy information into the speech generation task and presented variance adaptors to make the generated audio expressive.…”