Fcl-Taco2: Towards Fast, Controllable and Lightweight Text-to-Speech Synthesis

Abstract

Sequence-to-sequence (seq2seq) learning has greatly improved text-to-speech (TTS) synthesis performance, but effective implementation on resource-restricted devices remains challenging as seq2seq models are usually computationally expensive and memory intensive. To achieve fast inference speed and small model size while maintain high-quality speech, we propose FCL-taco2, a Fast, Controllable and Lightweight (FCL) TTS model based on Tacotron2. FCL-taco2 adopts a novel semi-autoregressive (SAR) mode for phoneme level based parallel mel-spectrograms generation conditioned on prosody features, leading to faster inference speed and higher prosody controllability than Tacotron2. Besides, knowledge distillation (KD) is leveraged to compress a relatively large FCL-taco2 model to its small version with minor loss of speech quality. Experimental results on English (EN) and Chinese (CN) datasets show that the small version of FCL-taco2 achieves comparable performance with Tacotron2 in terms of speech quality, while it has a 4.8 smaller footprint with 17.7 and 18.5 faster inference speeds on average for EN and CN experiments respectively. Besides, execution on mobile devices shows that the proposed model can achieve faster than real-time speech synthesis. Our code and audio samples are released 1 .

Controllable speech synthesis by learning discrete phoneme-level prosodic representations

“…Extending to other prosodic features, FastPitch [21] incorporates pitch control by also predicting F0 contours and FastPitchFormant [22] utilizes the predicted F0 in an excitation generator inspired by the source-filter theory in order to provide more robust and accurate pitch control. Since TTS decoders are conditioned on phoneme encoder representations, in FastSpeech 2 [23] and FCL-Taco2 [24] prosody prediction modules are introduced, which add prosodic information to these representations and are trained in a supervised manner utilizing ground truth values. In these cases, prosody information can be represented in various ways.…”

A Survey on Neural Speech Synthesis

“…It speeds up the Parallel WaveGAN vocoder by 4x without any degradation in sound quality. Wang et al [367] propose a semi-autoregressive mode for mel-spectrogram generation, where the melspectrograms are generated in an autoregressive mode for individual phoneme and non-autoregressive mode for different phonemes.…”

“…Speedup with Domain Knowledge Domain knowledge from speech can be leveraged to speed up inference, such as linear prediction [357], multiband modeling [411,400], subscale prediction [147], multi-frame prediction [420,376,367,123], streaming synthesis [74], etc. LPCNet [357] combines digital signal processing with neural networks, by using linear prediction coefficients to calculate the next waveform and a lightweight model to predict the residual value, which can speed the inference of autoregressive waveform generation.…”

Empathic Machines: Using Intermediate Features as Levers to Emulate Emotions in Text-To-Speech Systems

“…The trick non-autoregressive models use to generate Mel frames in parallel is to predict the relevant features as an intermediate step and condition the independent decoding of Mels on them. This technique is now increasingly adopted for au-toregressive models as well (Wang et al, 2021) to predict features like phoneme duration that improve decoding stability avoiding alignment issues. Our method is compatible with any architecture that predicts prosodic features of pitch, energy, and duration as an intermediate step before decoding.…”