BATUDE: Budget-Aware Neural Network Compression Based on Tucker Decomposition

Yin, Miao; Phan, Huy; Zang, Xiao; Liao, Siyu; Yuan, Bo

doi:10.1609/aaai.v36i8.20869

Cited by 9 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…And when dealing with high-order tensors, there will be too many auxiliary variables used in the augmented Lagrangian algorithm, which will affect convergence. Without using nuclear norm, [54] just set the upper bound of ranks, therefore it is applicable to various tensor decompositions.…”

Section: Compression-aware Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Low Rank Optimization for Efficient Deep Learning: Making a Balance Between Compact Architecture And Fast Training

Ou,

Chen,

Zhu

et al. 2024

J. of Syst. Eng. Electron.

View full text Add to dashboard Cite

Deep neural networks (DNNs) have achieved great success in many data processing applications. However, high computational complexity and storage cost make deep learning difficult to be used on resource-constrained devices, and it is not environmental-friendly with much power cost. In this paper, we focus on low-rank optimization for efficient deep learning techniques. In the space domain, DNNs are compressed by low rank approximation of the network parameters, which directly reduces the storage requirement with a smaller number of network parameters. In the time domain, the network parameters can be trained in a few subspaces, which enables efficient training for fast convergence. The model compression in the spatial domain is summarized into three categories as pre-train, pre-set, and compression-aware methods, respectively. With a series of integrable techniques discussed, such as sparse pruning, quantization, and entropy coding, we can ensemble them in an integration framework with lower computational complexity and storage. In addition to summary of recent technical advances, we have two findings for motivating future works. One is that the effective rank, derived from the Shannon entropy of the normalized singular values, outperforms other conventional sparse measures such as the norm for network compression. The other is a spatial and temporal balance for tensorized neural networks. For accelerating the training of tensorized neural networks, it is crucial to leverage redundancy for both model compression and subspace training.

show abstract

Section: Compression-aware Methodsmentioning

confidence: 99%

“…Recently, researches on compression-aware method emerge in large numbers and plenty of experiments show that with the premise of using the same tensor decomposition method, compression-aware method can outperform the other two methods [54,55,85]. Hence, we should pay more attention to it.…”

Section: Sparsity Measurementioning

confidence: 99%

Low Rank Optimization for Efficient Deep Learning: Making a Balance Between Compact Architecture And Fast Training

Ou,

Chen,

Zhu

et al. 2024

J. of Syst. Eng. Electron.

View full text Add to dashboard Cite

show abstract

“…Automatic Rank Selection. Our work is most closely related to (Gusak et al 2019;Liebenwein et al 2021;Li et al 2022;Yin et al 2022a), the state-of-the-art automatic rank selection solutions. Specifically, (Gusak et al 2019) proposes to utilize variational Bayesian matrix factorization to determine the ranks of the tensor decomposed DNNs in a multi-stage way.…”

Section: Related Workmentioning

confidence: 96%

HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks

Xiao

Zhang

Gong

et al. 2023

AAAI

View full text Add to dashboard Cite

Low-rank compression is an important model compression strategy for obtaining compact neural network models. In general, because the rank values directly determine the model complexity and model accuracy, proper selection of layer-wise rank is very critical and desired. To date, though many low-rank compression approaches, either selecting the ranks in a manual or automatic way, have been proposed, they suffer from costly manual trials or unsatisfied compression performance. In addition, all of the existing works are not designed in a hardware-aware way, limiting the practical performance of the compressed models on real-world hardware platforms. To address these challenges, in this paper we propose HALOC, a hardware-aware automatic low-rank compression framework. By interpreting automatic rank selection from an architecture search perspective, we develop an end-to-end solution to determine the suitable layer-wise ranks in a differentiable and hardware-aware way. We further propose design principles and mitigation strategy to efficiently explore the rank space and reduce the potential interference problem. Experimental results on different datasets and hardware platforms demonstrate the effectiveness of our proposed approach. On CIFAR-10 dataset, HALOC enables 0.07% and 0.38% accuracy increase over the uncompressed ResNet-20 and VGG-16 models with 72.20% and 86.44% fewer FLOPs, respectively. On ImageNet dataset, HALOC achieves 0.9% higher top-1 accuracy than the original ResNet-18 model with 66.16% fewer FLOPs. HALOC also shows 0.66% higher top-1 accuracy increase than the state-of-the-art automatic low-rank compression solution with fewer computational and memory costs. In addition, HALOC demonstrates the practical speedups on different hardware platforms, verified by the measurement results on desktop GPU, embedded GPU and ASIC accelerator.

show abstract

“…Miao et al [36] recently proposed a budget-aware rank selection method that can calculate tensor ranks via one-shot training. Although it can automatically select the proper tensor ranks for each layer, it may obtain different ranks when the training environment changes, such as the dataset and training hyperparameters.…”

Section: Related Workmentioning

confidence: 99%

Remote Sensing Imagery Object Detection Model Compression via Tucker Decomposition

Huyan

Jiang

et al. 2023

Mathematics

View full text Add to dashboard Cite

Although convolutional neural networks (CNNs) have made significant progress, their deployment onboard is still challenging because of their complexity and high processing cost. Tensors provide a natural and compact representation of CNN weights via suitable low-rank approximations. A novel decomposed module called DecomResnet based on Tucker decomposition was proposed to deploy a CNN object detection model on a satellite. We proposed a remote sensing image object detection model compression framework based on low-rank decomposition which consisted of four steps, namely (1) model initialization, (2) initial training, (3) decomposition of the trained model and reconstruction of the decomposed model, and (4) fine-tuning. To validate the performance of the decomposed model in our real mission, we constructed a dataset containing only two classes of objects based on the DOTA and HRSC2016. The proposed method was comprehensively evaluated on the NWPU VHR-10 dataset and the CAST-RS2 dataset created in this work. The experimental results demonstrated that the proposed method, which was based on Resnet-50, could achieve up to 4.44 times the compression ratio and 5.71 times the speedup ratio with merely a 1.9% decrease in the mAP (mean average precision) of the CAST-RS2 dataset and a 5.3% decrease the mAP of the NWPU VHR-10 dataset.

show abstract

BATUDE: Budget-Aware Neural Network Compression Based on Tucker Decomposition

Cited by 9 publications

References 25 publications

Low Rank Optimization for Efficient Deep Learning: Making a Balance Between Compact Architecture And Fast Training

Low Rank Optimization for Efficient Deep Learning: Making a Balance Between Compact Architecture And Fast Training

HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks

Remote Sensing Imagery Object Detection Model Compression via Tucker Decomposition

Contact Info

Product

Resources

About