Compression of recurrent neural networks for efficient language modeling

Abstract

Recurrent neural networks have proved to be an effective method for statistical language modeling. However, in practice their memory and run-time complexity are usually too large to be implemented in real-time offline mobile applications. In this paper we consider several compression techniques for recurrent neural networks including Long-Short Term Memory models. We make particular attention to the high-dimensional output problem caused by the very large vocabulary size. We focus on effective compression methods in the context of their exploitation on devices: pruning, quantization, and matrix decomposition approaches (low-rank factorization and tensor train decomposition, in particular). For each model we investigate the trade-off between its size, suitability for fast inference and perplexity. We propose a general pipeline for applying the most suitable methods to compress recurrent neural networks for language modeling. It has been shown in the experimental study with the Penn Treebank (PTB) dataset that the most efficient results in terms of speed and compression-perplexity balance are obtained by matrix decomposition techniques.

Efficient Language Modeling with Automatic Relevance Determination in Recurrent Neural Networks

“…We also compared our approach to other compression techniques: matrix decompositionbased (Grachev et al, 2019) and VDbased (Chirkova et al, 2018). For the last one we used a similar model: a network with one LSTM layer of 256 hidden units.…”

“…The weight gradually increases from zero to one during the first several epochs of training. This technique allows achieving better final performance of the model because such a train- 13.4 / 0.14 32.1% / 97.8% 91.84 27.2% LR for Softmax, (Grachev et al, 2019) 14.5 / 1.19 26.8 % / 81.7 % 84.12 N/A TT for Softmax, (Grachev et al, 2019) 14. (Chirkova et al, 2018) 3.2 / 0.12 43.3 % / 95.5 % 109.2 N/A DSVI-ARD (Ours)…”

“…Most of the works on neural networks compression can be roughly divided into two categories: those dealing with matrix decomposition approaches (Lu et al, 2016;Arjovsky et al, 2016;Tjandra et al, 2017;Grachev et al, 2019) and those that leverage pruning techniques (Han et al, 2015;Narang et al, 2017). From this point of view methods based on Bayesian techniques (Louizos et al, 2017;Molchanov et al, 2017) can be considered as a more mathematically justified version of pruning.…”

Long short‐term memory networks for proton dose calculation in highly heterogeneous tissues

“…Our study concentrated on the ability to estimate dose in heterogeneous geometries, and no effort was made in improving the model efficiency. Various model compression techniques, for example, pruning, quantization, and tensor decomposition methods (achieving low-rank structures in the weight matrices), [51][52][53] may substantially lower the number of parameters in fully connected layers. 54,55 The efficiency of the model can be further enhanced through fine-tuning of the model architecture.…”

Compression of Deep Learning Models for NLP

“…-Low rank factorization: [10,36] -Factorized embedding parameterization: [19] -Block-Term tensor decomposition: [23,38] -Singular Value Decomposition: [37] -Joint factorization of recurrent and inter-layer weight matrices: [28] -Tensor train decomposition: [10,17] -Sparse factorization: [6] • [11] • Applications: In this section, we will discuss application and success of various model compression methods across various popular NLP tasks like Language modeling, Machine translation, Summarization, Sentiment analysis, Question answering, Natural language inference, Paraphrasing, Image captioning, Handwritten character recognition. • Summary and future trends.…”