Investigation of Using Continuous Representation of Various Linguistic Units in Neural Network Based Text-to-Speech Synthesis

Abstract

SUMMARYBuilding high-quality text-to-speech (TTS) systems without expert knowledge of the target language and/or time-consuming manual annotation of speech and text data is an important yet challenging research topic. In this kind of TTS system, it is vital to find representation of the input text that is both effective and easy to acquire. Recently, the continuous representation of raw word inputs, called "word embedding", has been successfully used in various natural language processing tasks. It has also been used as the additional or alternative linguistic input features to a neural-network-based acoustic model for TTS systems. In this paper, we further investigate the use of this embedding technique to represent phonemes, syllables and phrases for the acoustic model based on the recurrent and feed-forward neural network. Results of the experiments show that most of these continuous representations cannot significantly improve the system's performance when they are fed into the acoustic model either as additional component or as a replacement of the conventional prosodic context. However, subjective evaluation shows that the continuous representation of phrases can achieve significant improvement when it is combined with the prosodic context as input to the acoustic model based on the feed-forward neural network.

Unsupervised Learning for Expressive Speech Synthesis

“…To name a few, use word embeddings to substitute TOBI and POS tags in RNN-based synthesis achieving significant system improvement. Wang et al (2016) enhance the input to NN-based systems with continuous word embeddings, and also try to substitute the conventional linguistic input by the word embeddings. They do not achieve performance improvement, however, when they use phrase embeddings combined with phonetic context, they do achieve significant improvement in a DNN-based system.…”

“…They do not achieve performance improvement, however, when they use phrase embeddings combined with phonetic context, they do achieve significant improvement in a DNN-based system. Wang et al (2016) enhances word vectors with prosodic information, i.e. updates them, achieving significant improvements.…”

Expressive Speech Synthesis Using Sentiment Embeddings

“…To name a few, Wang et al [13] use word embeddings to substitute TOBI and POS tags in RNN-based synthesis achieving significant system improvement. Wang et al [14] enhance the input to NN-based systems with continuous word embeddings, and also try to substitute the conventional linguistic input by the word embeddings. They do not achieve performance improvement, however, when they use phrase embeddings combined with phonetic context, they do achieve significant improvement in a DNN-based system.…”

“…They do not achieve performance improvement, however, when they use phrase embeddings combined with phonetic context, they do achieve significant improvement in a DNN-based system. Wang et al [14] enhance word vectors with prosodic information, i.e. update them, achieving significant improvements.…”

Enhance the Word Vector with Prosodic Information for the Recurrent Neural Network Based TTS System

“…Although word vectors are shown to be effective in various natural language processing tasks [8], implicit linguistic regularity encoded in these vectors may still be insufficient and noisy for the TTS task. Typically, our previous experiments implied that, at least on the utilized speech corpus, word vectors were not significantly better than the automatically derived prosodic symbols for TTS systems with a acoustic model based on either the recurrent neural network (RNN) or the deep feedforward neural network (DNN) [9].…”