Compression of convolutional neural networks: A short survey

Pilipović, Ratko; Bulić, Patricio; Risojević, Vladimir

doi:10.1109/infoteh.2018.8345545

Cited by 8 publications

(10 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first weight pruning method i.e. Optimal Brain Damage (OBD) is introduced in [10]. It removes weights based on their saliency measure in an iterative manner.…”

Section: A Network Compression Using Pruning Methodsmentioning

confidence: 99%

Resource-Restricted Environments Based Memory-Efficient Compressed Convolutional Neural Network Model for Image-Level Object Classification

et al. 2023

View full text Add to dashboard Cite

In the past decade, Convolutional Neural Networks (CNNs) have achieved tremendous success in solving complex classification problems. CNN architectures require an excessive number of computations to achieve high accuracy. However, these models are deficient due to the heavy cost of storage and energy, which prohibits the application of CNNs to resource-constrained edge-devices. Hence, developing aggressive optimization schemes for efficient deployment of CNNs on edge devices has become the most important requirement. To find the optimal approach, we present a resource-limited environment based memoryefficient network compression model for image-level object classification. The main aim is to compress CNN architecture by achieving low computational cost and memory requirements without dropping system's accuracy. To achieve the said goal, we propose a network compression strategy, that works in a collaborative manner, where Soft Filter Pruning is first applied to reduce the computational cost of the model. In the next step, the model is divided into No-Pruning Layers (NP-Layers) and Pruning Layers (P-Layers). Incremental Quantization is applied to P-Layers due to irregular weights distribution, while for NP-Layers, we propose a novel Optimized Quantization algorithm for the quantization of weights up to optimal levels obtained from the Optimizer. This scheme is designed to achieve the best trade-off between compression ratio and accuracy of the model. Our proposed system is validated for image-level object classification on LeNet-5, CIFARquick, and VGG-16 networks using MNIST, CIFAR-10, and ImageNet ILSVRC2012 datasets respectively. We have achieved high compression ratio with negligible accuracy drop, outperforming the state-of-the-art methods.

show abstract

“…The first weight pruning method i.e. Optimal Brain Damage (OBD) is introduced in [10]. It removes weights based on their saliency measure in an iterative manner.…”

Section: A Network Compression Using Pruning Methodsmentioning

confidence: 99%

Resource-Restricted Environments Based Memory-Efficient Compressed Convolutional Neural Network Model for Image-Level Object Classification

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Other networks like BinaryConnect [36] or BinaryNet [37] directly train models with 1-bit integer weights and are a type of deep neural networks in themselves called Binarized Neural Networks or BNNs. Another one of the most appealing advantages of this method is that accuracy can be maintained or if not, the accuracy drop is in most cases less than 1% with respect to the original one [38]. There are two main types of quantization: linear or uniform and non-linear or non-uniform [39].…”

Section: Quantizationmentioning

confidence: 99%

“…The four methods explained in prior subsections compress deep neural networks and reduce the number of parameters, but their architecture, the number and orders of the layers are remaining the same. A way to reduce the weights in a CNN is using filter decomposition which consists of replacing big convolutional layers by more convolution layers and smaller filters [38].…”

Section: Design Of Smaller Architecturesmentioning

confidence: 99%

See 1 more Smart Citation

Accuracy-aware Structured Filter Pruning for Deep Neural Networks

Carballo

Lee

2020

2020 International Conference on Computational Science and Computational Intelligence (CSCI)

View full text Add to dashboard Cite

Deep neural networks (DNNs) have several technical issues on computational complexity, redundancy, and the parameter sizeespecially when applied in embedded devices. Among those issues, lots of parameters require high memory capacity which causes migration problem to embedded devices. Many pruning techniques are proposed to reduce the network size in deep neural networks, but there are still various issues that exist for applying pruning techniques to DNNs. In this paper, we propose a simple-yetefficient scheme, accuracy-aware structured pruning based on the characterization of each convolutional layer. We investigate the accuracy and compression rate of individual layer with a fixed pruning ratio and re-order the pruning priority depending on the accuracy of each layer. To achieve a further compression rate, we also add quantization to the linear layers. Our results show that the order of the layers pruned does affect the final accuracy of the deep neural network. Based on our experiments, the pruned AlexNet and VGG16 models' parameter size is compressed up to 47.28x and 35.21x with less than 1% accuracy drop with respect to the original model.

show abstract

“…Hardware accelerators [141,192] are designed primarily for network acceleration. They include specialized cen-CNN Acceleration [39,38,133,128,184,253,172] Network Optimization…”

Section: Introductionmentioning

confidence: 99%

Pruning and Quantization for Deep Neural Network Acceleration: A Survey

Liang

Glossner

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks have been applied in many applications exhibiting extraordinary abilities in the field of computer vision. However, complex network architectures challenge efficient real-time deployment and require significant computation resources and energy costs. These challenges can be overcome through optimizations such as network compression. This paper provides a survey on two types of network compression: pruning and quantization. We compare current techniques, analyze their strengths and weaknesses, provide guidance for compressing networks, and discuss possible future compression techniques.

show abstract

Compression of convolutional neural networks: A short survey

Cited by 8 publications

References 7 publications

Resource-Restricted Environments Based Memory-Efficient Compressed Convolutional Neural Network Model for Image-Level Object Classification

Resource-Restricted Environments Based Memory-Efficient Compressed Convolutional Neural Network Model for Image-Level Object Classification

Accuracy-aware Structured Filter Pruning for Deep Neural Networks

Pruning and Quantization for Deep Neural Network Acceleration: A Survey

Contact Info

Product

Resources

About