Adversarially Trained End-to-End Korean Singing Voice Synthesis System

Abstract

In this paper, we propose an end-to-end Korean singing voice synthesis system from lyrics and a symbolic melody using the following three novel approaches: 1) phonetic enhancement masking, 2) local conditioning of text and pitch to the superresolution network, and 3) conditional adversarial training. The proposed system consists of two main modules; a mel-synthesis network that generates a mel-spectrogram from the given input information, and a super-resolution network that upsamples the generated mel-spectrogram into a linear-spectrogram. In the mel-synthesis network, phonetic enhancement masking is applied to generate implicit formant masks solely from the input text, which enables a more accurate phonetic control of singing voice. In addition, we show that two other proposed methods -local conditioning of text and pitch, and conditional adversarial training -are crucial for a realistic generation of the human singing voice in the super-resolution process. Finally, both quantitative and qualitative evaluations are conducted, confirming the validity of all proposed methods.

Fast and High-Quality Singing Voice Synthesis System Based on Convolutional Neural Networks

“…In recent years, several kinds of DNN-based singing voice synthesis systems [4,17,18,19,20] have been proposed. In the training part of the basic system [4], parameters for spectrum (e.g., melcepstral coefficients), excitation, and vibrato are extracted from a singing voice database as acoustic features.…”

Disentangling Timbre and Singing Style with Multi-Singer Singing Synthesis System

“…We propose a multi-singer SVS system that can model timbre and singing styles independently. We designed the network with [8] as the baseline and extended the existing model to the multi-singer model by adding 1) singer identity encoder and 2) timbre/singing style conditioning method. As shown in Fig.…”

“…Finally, to create a linear spectrogram that is more realistic, we applied adversarial training and added a discriminator to this end. Please refer to [8] for more detailed information on each module of the network. The summary of the generation process of the entire network is as follows:…”

“…To this end, we propose a conditioning method that can model timbre and singing styles, respectively, while extending our existing single-singer SVS system [8] to a multisinger system. First, we add a singer identity encoder to the baseline model to capture the singer's global identity.…”

Mellotron: Multispeaker Expressive Voice Synthesis by Conditioning on Rhythm, Pitch and Global Style Tokens

“…By accounting for melodic information such as pitch and rhythm, expressive speech synthesis with Mellotron can be easily extended to singing voice synthesis (SVS) [3,4]. Unfortunately, recent attempts [4] require a singing voice dataset and heavily quantized pitch and rhythm data obtained from a digital representation of a music score, for example MIDI [5] or musicXML [6]. Mellotron does not require any singing voice in the dataset nor manually aligned pitch and text in order to synthesize singing voice.…”