Heuristic Rank Selection with Progressively Searching Tensor Ring Network

Li, Nannan; Pan, Yu; Chen, Yaran; Ding, Zixiang; Zhao, Dan; Xu, Zenglin

doi:10.48550/arxiv.2009.10580

Cited by 2 publications

(17 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, TT and TR approaches do not perform well on CNN models. For instance, even the very recent progress [43,32] still suffers 1.0% accuracy loss with 2.7ĉ ompression ratio, or even 1.9% accuracy loss with 5.8ĉ ompression ratio, both for ResNet-32 model on CIFAR-10 dataset. From the perspective of practical deployment, such non-negligible accuracy degradation severely hinders the widespread adoption of tensor decomposition for many CNN-involved model compression scenarios.…”

Section: Related Work On Dnn Model Compressionmentioning

confidence: 99%

“…Uncompressed -99.21 1.0Ŝ tandard TR [43] TR 99.10 10.5P STRN-M [32] 99.43 16.5P STRN-S [32] 99.51 6.5Ŝ tandard TT [12] TT 99.07 17.9Ô urs 99.48 17.9Ô urs 99.51 8.3T able 1: LeNet-5 on MNIST dataset using different TT/TRformat compression approaches.…”

Section: Comp Ratiomentioning

confidence: 99%

“…ResNet-20 Uncompressed -91.25 1.0Ŝ tandard TR [43,32] TR 87.5 5.4T R-RL [5] 88.3 6.8P STRN-M [32] 88.50 6.8P STRN-S [32] 90.80 2.5Ŝ tandard TT [12] TT ADMM-based TT-format model with the uncompressed model as well as the state-of-the-art TT/TR-format works. It is seen that our ADMM-based compression can achieve the highest compression ratio and the best accuracy.…”

Section: Comp Ratiomentioning

confidence: 99%

“…ResNet-20 Uncompressed -65.4 1.0Ŝ tandard TR [43,32] TR 63.55 4.7P STRN-M [32] 63.62 4.7P STRN-S [32] 66.13 2.3Ŝ tandard TT [12] TT [43] TR 66.70 4.8P STRN-M [32] 66.77 5.2P STRN-S [32] 68.05 2.4Ŝ tandard TT [12,43] TT 62.90 4.6Ô urs 67.17 For ResNet-32, again, standard training on compressed models using TT or TR decomposition causes huge performance degradation. The state-of-the-art PSTRN-S/M indeed brings performance improvement, but the test accuracy is still not satisfied.…”

Section: Comp Ratiomentioning

confidence: 99%

“…For compressing convolutional neural network (CNN) in the image classification task, which are the most commonly used and representa-tive setting for evaluating model compression performance, all the state-of-the-art tensor decomposition approaches, including TT and TR, suffer very significant accuracy loss. For instance, even the very recent progress [32] using TR still has 1.0% accuracy loss when the compression ratio is only 2.7ˆfor ResNet-32 model on CIFAR-10 dataset. For the larger compression ratio as 5.8ˆ, the accuracy loss further increases to 1.9% .…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework

Yin¹,

Yang²,

Liao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Advanced tensor decomposition, such as tensor train (TT) and tensor ring (TR), has been widely studied for deep neural network (DNN) model compression, especially for recurrent neural networks (RNNs). However, compressing convolutional neural networks (CNNs) using TT/TR always suffers significant accuracy loss. In this paper, we propose a systematic framework for tensor decomposition-based model compression using Alternating Direction Method of Multipliers (ADMM). By formulating TT decompositionbased model compression to an optimization problem with constraints on tensor ranks, we leverage ADMM technique to systemically solve this optimization problem in an iterative way. During this procedure, the entire DNN model is trained in the original structure instead of TT format, but gradually enjoys the desired low tensor rank characteristics. We then decompose this uncompressed model to TT format, and fine-tune it to finally obtain a high-accuracy TTformat DNN model. Our framework is very general, and it works for both CNNs and RNNs, and can be easily modified to fit other tensor decomposition approaches. We evaluate our proposed framework on different DNN models for image classification and video recognition tasks. Experimental results show that our ADMM-based TT-format models demonstrate very high compression performance with high accuracy. Notably, on CIFAR-100, with 2.3ˆand 2.4ˆcompression ratios, our models have 1.96% and 2.21% higher top-1 accuracy than the original ResNet-20 and ResNet-32, respectively. For compressing ResNet-18 on ImageNet, our model achieves 2.47ˆFLOPs reduction without accuracy loss.

show abstract

Section: Related Work On Dnn Model Compressionmentioning

confidence: 99%

Section: Comp Ratiomentioning

confidence: 99%

Section: Comp Ratiomentioning

confidence: 99%

Section: Comp Ratiomentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework

Yin¹,

Yang²,

Liao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

BATUDE: Budget-Aware Neural Network Compression Based on Tucker Decomposition

Yin

Phan

Zang

et al. 2022

AAAI

View full text Add to dashboard Cite

Model compression is very important for the efficient deployment of deep neural network (DNN) models on resource-constrained devices. Among various model compression approaches, high-order tensor decomposition is particularly attractive and useful because the decomposed model is very small and fully structured. For this category of approaches, tensor ranks are the most important hyper-parameters that directly determine the architecture and task performance of the compressed DNN models. However, as an NP-hard problem, selecting optimal tensor ranks under the desired budget is very challenging and the state-of-the-art studies suffer from unsatisfied compression performance and timing-consuming search procedures. To systematically address this fundamental problem, in this paper we propose BATUDE, a Budget-Aware TUcker DEcomposition-based compression approach that can efficiently calculate optimal tensor ranks via one-shot training. By integrating the rank selecting procedure to the DNN training process with a specified compression budget, the tensor ranks of the DNN models are learned from the data and thereby bringing very significant improvement on both compression ratio and classification accuracy for the compressed models. The experimental results on ImageNet dataset show that our method enjoys 0.33% top-5 higher accuracy with 2.52X less computational cost as compared to the uncompressed ResNet-18 model. For ResNet-50, the proposed approach enables 0.37% and 0.55% top-5 accuracy increase with 2.97X and 2.04X computational cost reduction, respectively, over the uncompressed model.

show abstract

Heuristic Rank Selection with Progressively Searching Tensor Ring Network

Cited by 2 publications

References 21 publications

Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework

Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework

BATUDE: Budget-Aware Neural Network Compression Based on Tucker Decomposition

Contact Info

Product

Resources

About