Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Abstract

We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called VALL-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. VALL-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that VALL-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find VALL-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.

Can an AI Win Ghana's National Science and Maths Quiz? An AI Grand Challenge for Education

“…It would need to appropriately say answers for different scientific and mathematical textual representations. A recent model such as Microsoft's VALL-E which can simulate a person's voice could be used Wang et al (2023).…”

Ubiq-Genie: Leveraging External Frameworks for Enhanced Social VR Experiences

“…Generative Models Incorporating other generative models in Ubiq-Genie, beyond the employed image and text synthesis models, could lead to interesting applications. Potential models to be integrated could be capable of synthesising 3D models from text or images such as Point-E [16], personalised speech from text such as VALL-E [24], or audio from text or images such as Make-An-Audio [9] and MusicLM [1]. In addition, the currently implemented services could be expanded to build more advanced types of applications and experiences.…”

SpeechPainter: Text-conditioned Speech Inpainting

“…VALL-E (Wang et al, 2023) instead relies on a hybrid approach, where the tokens corresponding to the first RVQ level are predicted autoregressively, and the subsequent levels are produced non-autoregressively. The latter is achieved by a model that sums up the embeddings from the same RVQ input frame, and applies bidirectional self-attention to predict all tokens from RVQ level q + 1 given all tokens from levels 1, .…”

“…Modeling discrete representations of audio produced by neural codecs (Zeghidour et al, 2022;Défossez et al, 2022) makes the task of audio generation amenable to the powerful Transformer-based sequence-to-sequence modeling approaches (Vaswani et al, 2017). Casting unconditional and conditional audio generation as sequence-to-sequence modeling has unlocked rapid progress in speech continuation (Borsos et al, 2022), text-to-speech (Wang et al, 2023;Kharitonov et al, 2023), and general audio and music generation (Kreuk et al, 2022;Agostinelli et al, 2023).…”

“…The problem of generating long audio token sequences can be addressed by at least three orthogonal approaches, or a combination thereof: i) efficient attention mechanisms (Kitaev et al, 2020;Choromanski et al, 2021;Xiong et al, 2021;Hawthorne et al, 2022), ii) non-autoregressive, parallel decoding schemes (Gu et al, 2017;Ghazvininejad et al, 2019;Chang et al, 2022), iii) custom architectures adapted to the special structure of the tokens produced by neural audio codecs (Kreuk et al, 2022;Wang et al, 2023;Lee et al, 2022). However, in the context of modeling the token sequence of neural audio codecs, either unconditionally or based on weak conditioning such as text, the efficient generation of long, high-quality audio segments remains an open problem.…”