Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices

Zen, Heiga; Agiomyrgiannakis, Yannis; Egberts, Niels; Henderson, Fergus; Szczepaniak, Przemysław

doi:10.48550/arxiv.1606.06061

Cited by 9 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another direction focuses on efficient storage and representation of weights. Various techniques, such as weight sharing within Toeplitz matrices [19], weight tying through effective hashing [20], and appropriate weight quantization [21][22][23], can greatly reduce model size, in some cases at the expense of a slight performance degradation.…”

Section: Related Workmentioning

confidence: 99%

Grow and Prune Compact, Fast, and Accurate LSTMs

Dai

Yin

Jha

2020

IEEE Trans. Comput.

View full text Add to dashboard Cite

Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-layer LSTM (H-LSTM) that adds hidden layers to LSTM's original onelevel non-linear control gates. H-LSTM increases accuracy while employing fewer external stacked layers, thus reducing the number of parameters and run-time latency significantly. We employ grow-and-prune (GP) training to iteratively adjust the hidden layers through gradient-based growth and magnitude-based pruning of connections. This learns both the weights and the compact architecture of H-LSTM control gates. We have GP-trained H-LSTMs for image captioning and speech recognition applications. For the NeuralTalk architecture on the MSCOCO dataset, our three models reduce the number of parameters by 38.7× [floating-point operations (FLOPs) by 45.5×], run-time latency by 4.5×, and improve the CIDEr score by 2.6. For the DeepSpeech2 architecture on the AN4 dataset, our two models reduce the number of parameters by 19.4× (FLOPs by 23.5×), run-time latency by 15.7%, and the word error rate from 12.9% to 8.7%. Thus, GP-trained H-LSTMs can be seen to be compact, fast, and accurate.

show abstract

Section: Related Workmentioning

confidence: 99%

Grow and Prune Compact, Fast, and Accurate LSTMs

Dai

Yin

Jha

2020

IEEE Trans. Comput.

View full text Add to dashboard Cite

show abstract

“…Since production speech synthesis systems have very low tolerance on such instability, end-to-end speech synthesis systems have not been widely deployed in practical applications. In DurIAN, we replace the attention mechanism with an alignment model [15,16], in which the alignment between the phoneme sequence and the target acoustic sequence is inferred from a phoneme duration prediction model. The duration of each phoneme is measured by the number of aligned acoustic frames.…”

Section: Alignment Modelmentioning

confidence: 99%

DurIAN: Duration Informed Attention Network For Multimodal Synthesis

Yu,

Lu,

et al. 2019

Preprint

View full text Add to dashboard Cite

In this paper, we present a generic and robust multimodal synthesis system that produces highly natural speech and facial expression simultaneously. The key component of this system is the Duration Informed Attention Network (DurIAN), an autoregressive model in which the alignments between the input text and the output acoustic features are inferred from a duration model. This is different from the end-to-end attention mechanism used, and accounts for various unavoidable artifacts, in existing end-to-end speech synthesis systems such as Tacotron. Furthermore, DurIAN can be used to generate high quality facial expression which can be synchronized with generated speech with/without parallel speech and face data. To improve the efficiency of speech generation, we also propose a multi-band parallel generation strategy on top of the WaveRNN model. The proposed Multi-band WaveRNN effectively reduces the total computational complexity from 9.8 to 3.6 GFLOPS, and is able to generate audio that is 6 times faster than real time on a single CPU core. We show that DurIAN could generate highly natural speech that is on par with current state of the art end-to-end systems, while at the same time avoid word skipping/repeating errors in those systems. Finally, a simple yet effective approach for fine-grained control of expressiveness of speech and facial expression is introduced.

show abstract

“…Non-pruning methods. In addition to pruning, other approaches also make significant contribution to LSTMs compression, including distillation (Tian et al, 2017), matrix factorization (Kuchaiev & Ginsburg, 2017), parameter sharing (Lu et al, 2016), group Lasso regularization (Wen et al, 2017), weight quantization (Zen et al, 2016), etc.…”

Section: Lstm Compressionmentioning

confidence: 99%

Intrinsically Sparse Long Short-Term Memory Networks

Liu,

Mocanu,

Pechenizkiy

2019

Preprint

View full text Add to dashboard Cite

Long Short-Term Memory (LSTM) has achieved state-of-the-art performances on a wide range of tasks. Its outstanding performance is guaranteed by the long-term memory ability which matches the sequential data perfectly and the gating structure controlling the information flow. However, LSTMs are prone to be memory-bandwidth limited in realistic applications and need an unbearable period of training and inference time as the model size is ever-increasing. To tackle this problem, various efficient model compression methods have been proposed. Most of them need a big and expensive pre-trained model which is a nightmare for resource-limited devices where the memory budget is strictly limited. To remedy this situation, in this paper, we incorporate the Sparse Evolutionary Training (SET) procedure into LSTM, proposing a novel model dubbed SET-LSTM. Rather than starting with a fully-connected architecture, SET-LSTM has a sparse topology and dramatically fewer parameters in both phases, training and inference. Considering the specific architecture of LSTMs, we replace the LSTM cells and embedding layers with sparse structures and further on, use an evolutionary strategy to adapt the sparse connectivity to the data. Additionally, we find that SET-LSTM can provide many different good combinations of sparse connectivity to substitute the overparameterized optimization problem of dense neural networks. Evaluated on four sentiment analysis classification datasets, the results demonstrate that our proposed model is able to achieve usually better performance than its fully connected counterpart while having less than 4% of its parameters.

show abstract

Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices

Cited by 9 publications

References 0 publications

Grow and Prune Compact, Fast, and Accurate LSTMs

Grow and Prune Compact, Fast, and Accurate LSTMs

DurIAN: Duration Informed Attention Network For Multimodal Synthesis

Intrinsically Sparse Long Short-Term Memory Networks

Contact Info

Product

Resources

About