ADA-Tucker: Compressing deep neural networks via adaptive dimension adjustment tucker decomposition

Abstract

Despite recent success of deep learning models in numerous applications, their widespread use on mobile devices is seriously impeded by storage and computational requirements. In this paper, we propose a novel network compression method called Adaptive Dimension Adjustment Tucker decomposition (ADA-Tucker). With learnable core tensors and transformation matrices, ADA-Tucker performs Tucker decomposition of arbitrary-order tensors. Furthermore, we propose that weight tensors in networks with proper order and balanced dimension are easier to be compressed. Therefore, the high flexibility in decomposition choice distinguishes ADA-Tucker from all previous low-rank models. To compress more, we further extend the model to Shared Core ADA-Tucker (SCADA-Tucker) by defining a shared core tensor for all layers. Our methods require no overhead of recording indices of non-zero elements. Without loss of accuracy, our methods reduce the storage of LeNet-5 and LeNet-300 by ratios of 691× and 233×, respectively, significantly outperforming state of the art. The effectiveness of our methods is also evaluated on other three benchmarks (CIFAR-10, SVHN, ILSVRC12) and modern newly deep networks (ResNet, Wide-ResNet). × × × × ≈ × Figure 1: Four categories of mainstream compression methods, which can be combined for higher compression ratio. "a×" etc. means that the network is compressed by a times. Reducing the bits of weight representation and Effective coding There are two approaches for the first category: clustering and quantization. BinaryConnect Courbariaux et al. (2015) enforces weights in neural networks to take binary values. Incremental network quantization Zhou et al. (2017) quantizes deep models with 5 bits incrementally. Gong et al. Gong et al. (2015) learn CNNs in advance, and then apply k-means clustering on the weights for quantization. Ullrich et al. Ullrich et al. (2017) cluster the weights with a Gaussian mixture model (GMM), using only six class centers to represent all weights. The second category, effective coding, always combines with the first category, where the coding scheme is mainly Huffman coding. DeepCompression Han et al. (2016) first introduces Huffman coding in network compression and improves the compression ratios further. CNNPack Wang et al. (2016) also uses Huffman coding and gets better results. Making weights sparse Sparsity can be induced in either the original domain or the frequency domain. The most commonly used sparsity method in the original domain is pruning. Han et al. Han et al. (2015) recursively train a neural network and prune unimportant connections based on their weight magnitude. Dynamic network surgery Guo et al. (2016) prunes and splices the branch of the network. The frequency domain sparsity methods benefit from discrete cosine transformation (DCT). Chen et al. Chen et al. (2016) take advantage of DCT to make weights sparse in the frequency domain. Wang et al. Wang et al. (2016) combine DCT, clustering and Huffman coding for further compression. Simplifying the network structur...

Compressing CNN by alternating constraint optimization framework

“…Compression methods based on pre-trained model mainly include indirect compression, which consist of three steps: pre-trained model, model compression and fine-tuning. Tucker-2 decomposition 10 , as a mathematical method to explore low-rank features of large-scale tensor data, has been widely used for indirect compression.…”

“…However, other compression methods, such as pruning 11 and knowledge distillation 12 , have demonstrated that proper utilization of information from pre-trained models is crucial for model compression 13 . For Tucker decomposition-based indirect compression 10 , although the information of the pre-trained model is utilized, the pre-trained model still lacks low-rank property. The approximation error after direct low-rank tensor decomposition is too large to properly recover even using fine-tuning.…”

Post-Training Quantization for Vision Transformer

“…Among various compression methods like pruning [16,29,19] and weight decomposition [36], quantization method [9,37,7,27,14] compresses a neural network by using lower bit-width for weight values without changing the model architecture, which is particularly useful for carefullydesigned network architectures like transformers. Quantizing both weights and inputs can speed up inference by tuning floating-point operations into integer or bit operations.…”

Compression and Interpretability of Deep Neural Networks via Tucker Tensor Layer: From First Principles to Tensor Valued Back-Propagation

“…Further research in the area, in particular involving the combination of the TKD with NNs, includes the work in [16] and [17]. In [16] the TKD was used as a convolution kernel for mobile applications, while the authors in [17] adopt a more theoretical approach and propose a method to adaptively adjust the dimensions of the weight tensor in a layer-wise fashion. Both methods have been shown to achieve significant NN parameter compression and consequent reduction in training time.…”

“…The authors of the seminal paper [12] are the only to have analytically re-derived back-propagation [18] in terms of the tensor factors stemming from the TT decomposition, however, the dimensionality and order of the employed TTs were arbitrarily chosen, so that, despite the achieved compression, the results were not physically inter-pretable. Most other research in the field tends to rely on automatic differentiation for back-propagation [13], [14], [16], [17]. Although this is convenient for implementation purposes, as automatic differentiation is quite efficient, there is still a necessity for a deep analytical understanding of how the errors are propagated; this is fundamental to the explainability of DNNs.…”