MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder

Abstract

Multi-speaker singing voice synthesis is to generate the singing voice sung by different speakers. To generalize to new speakers, previous zero-shot singing adaptation methods obtain the timbre of the target speaker with a fixed-size embedding from single reference audio. However, they face several challenges: 1) the fixed-size speaker embedding is not powerful enough to capture full details of the target timbre; 2) single reference audio does not contain sufficient timbre information of the target speaker; 3) the pitch inconsistency between different speakers also leads to a degradation in the generated voice. In this paper, we propose a new model called MR-SVS to tackle these problems. Specifically, we employ both a multi-reference encoder and a fixed-size encoder to encode the timbre of the target speaker from multiple reference audios. The Multi-reference encoder can capture more details and variations of the target timbre. Besides, we propose a well-designed pitch shift method to address the pitch inconsistency problem. Experiments indicate that our method outperforms the baseline method both in naturalness and similarity. 1

MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity

“…Multi-singer SVS In the speech synthesis fields, many multi-speaker TTS models reflect speaker ID by feeding a fixed-size speaker embedding into the decoder [34,35]. Most multi-singer SVS models represent singer characteristics in similar ways [31,36,37]. However, learning the characteristics requires sufficient data for each singer.…”

“…[38] learns the characteristics of each singer by adapting a pre-trained SVS model to a target singer. [36,37] present zero-shot style adaptation methods that apply reference encoders that extract singer embeddings from the reference audio. In particular, [37] applies multiple reference encoders and multi-head attention to reflect singer characteristics more effectively.…”

“…[36,37] present zero-shot style adaptation methods that apply reference encoders that extract singer embeddings from the reference audio. In particular, [37] applies multiple reference encoders and multi-head attention to reflect singer characteristics more effectively. On the other hand, [39] proposes a method to disentangle timbre from singing style and control them separately.…”

PHONEix: Acoustic Feature Processing Strategy for Enhanced Singing Pronunciation with Phoneme Distribution Predictor

“…To further distinguish vowels and consonants, a duration predictor is built to produce fine-grained *Corresponding author. phoneme-level duration, which is trained based on supervision calculated by force-alignment [6][7][8][9][10][11], heuristics [12][13][14][15] etc. The advantage of this type of feature processing strategy is that the input phoneme and pitch sequence are strictly aligned at the note level based on the music score.…”