Sequence-To-Sequence Singing Voice Synthesis With Perceptual Entropy Loss

Abstract

The neural network (NN) based singing voice synthesis (SVS) systems require sufficient data to train well and are are prone to over-fitting due to data scarcity. However, we often encounter data limitation problem in building SVS systems because of high data acquisition and annotation cost, . In this work, we propose a Perceptual Entropy (PE) loss derived from a psycho-acoustic hearing model to regularize the network. With a one-hour open-source singing voice database, we explore the impact of the PE loss on various mainstream sequence-to-sequence models, including the RNN-based, transformer-based, and conformer-based models. Our experiments show that the PE loss can mitigate the over-fitting problem and significantly improve the synthesized singing quality reflected in objective and subjective evaluations.

PHONEix: Acoustic Feature Processing Strategy for Enhanced Singing Pronunciation with Phoneme Distribution Predictor

“…The experiments are conducted using the music processing toolkit Muskits [28], which is adapted from ESPnet [30]. More details about the codes can be found on github 2 .…”

“…To be specific, we utilize 384-dimensional embedding layers for lyrics, notes, and note durations. The encoder and decoder of the first model are both three-layer 256-dimensional bidirectional Long-Short Term Memory units (Bi-LSTM) following [2]. The second network utilizes a Transformer structure from [14].…”

“…The second network utilizes a Transformer structure from [14]. Apart from the additional loss for phoneme distribution predictor, we apply the same loss functions as [2] and [14]. We use the same HiFi-GAN vocoder [32] to convert the predicted acoustic features into waveforms.…”

“…SVS aims to generate a singing voice from a music score that contains pitch and duration information organized by notes with corresponding lyrics [1]. Different from the similar task of text-tospeech (TTS), it is difficult to obtain large public singing datasets for SVS [2][3][4], due to the copyright restrictions and the strict recording environment requirements. Moreover, singing has higher variability than spoken language, as singers have the flexibility to make changes to scores, making singing more natural and pleasing.…”

SUSing: SU-net for Singing Voice Synthesis

“…Ren et al [28] used a forward transformerbased network to perform end-to-end singing voice synthesis, directly generating a linear spectrum and then obtaining the singing voice through the Griffin-Lim vocoder. Shi et al [29] combined the perceptual entropy loss function with mainstream time sequence models, including RNN, transformer, and conformer for singing voice synthesis. Xue et al [30] used an acoustic model of the encoder-decoder architecture to perform end-to-end training on frame-level input.…”