Compressing recurrent neural network with tensor train

Abstract

Recurrent Neural Network (RNN) are a popular choice for modeling temporal and sequential tasks and achieve many state-of-the-art performance on various complex problems. However, most of the state-of-the-art RNNs have millions of parameters and require many computational resources for training and predicting new data. This paper proposes an alternative RNN model to reduce the number of parameters significantly by representing the weight parameters based on Tensor Train (TT) format. In this paper, we implement the TT-format representation for several RNN architectures such as simple RNN and Gated Recurrent Unit (GRU). We compare and evaluate our proposed RNN model with uncompressed RNN model on sequence classification and sequence prediction tasks. Our proposed RNNs with TT-format are able to preserve the performance while reducing the number of RNN parameters significantly up to 40 times smaller.

Compression of recurrent neural networks for efficient language modeling

“…Different matrix decomposition techniques can be also related to this class of compression methods. These methods can be as simple as low-rank decomposition or more complex like Tensor Train (TT) decomposition [24,25,26,27]. However, the TT-based approach have not been studied in language modeling task, where there are such issues as highdimensional input and output data, and, as a consequence, more options to configure TT decomposition.…”

“…This approach was successfully applied to compress fully connected neural networks [24], to develop convolutional TT layer [25] and to compress and improve RNNs [26,27]. However, there are still no studies of the TT decomposition for language modeling and similar tasks with high-dimensional outputs at the softmax layer.…”

Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models

“…A common challenge of the above technique is to determine the tensor rank. Exactly determining a tensor rank in general [49]- [51] Tensor Train [49] CP [55] Tucker [53], [55], [57] Tensor Train [52], [54], [55] CP [48] Tucker Tensor Train is NP-hard [47]. Therefore, in practice one often leverages numerical optimization or statistical techniques to obtain a reasonable rank estimation.…”

“…In order to avoid the expensive pre-training in the uncompressed format, the work in [52] and [53] directly trained fully connected and convolution layers in low-rank tensor-train and Tucker format with the tensor ranks fixed in advance. This idea has also been applied to recurrent neural networks [54], [55].…”

Bayesian Compression for Natural Language Processing

“…There are a lot of RNNs compression methods based on specific weight matrix representations (Tjandra et al, 2017;Le et al, 2015) or sparsification (Narang et al, 2017;Wen et al, 2018). In this paper we focus on RNNs compression via sparsification.…”