Variational Convolutional Neural Network Pruning

Zhao, Chenglong; Ni, Bingbing; Zhang, Jian; Zhang, Qiwei; Zhang, Wenjun; Tian, Qi

doi:10.1109/cvpr.2019.00289

Cited by 256 publications

(144 citation statements)

References 17 publications

Supporting

Mentioning

141

Contrasting

Unclassified

Order By: Relevance

“…"smaller-norm-lessimportant" is also an early approach based on a distancebased criterion [20], [21]. Recent studies have proposed new criteria-based methods using variational information [34] and geometric median [22] to overcome the distance-based criterion. Lastly, recent studies have proposed a clusteringbased approach to in depth analyze the similarity among filters [19], [23].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Filter Pruning and Re-Initialization via Latent Space Clustering

Lee

Heo

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Filter pruning is prevalent for pruning-based model compression. Most filter pruning methods have two main issues: 1) the pruned network capability depends on that of source pretrained models, and 2) they do not consider that filter weights follow a normal distribution. To address these issues, we propose a new pruning method employing both weight re-initialization and latent space clustering. For latent space clustering, we define filters and their feature maps as vertices and edges to be a graph, transformed into a latent space by graph convolution, alleviating to prune zero-near weight filters only. In addition, a part of filters is re-initialized with a constraint for enhancing filter diversity, and thus the pruned model is less dependent on the source network. This approach provides more robust accuracy even when pruned from the pretrained model with low accuracy. Extensive experimental results show our method decreases 56.6% and 84.6% of FLOPs and parameters of VGG16 with negligible loss of accuracy on CIFAR100, which is the state-of-the art performance. Furthermore, our method presents outperforming or comparable pruning results against state-of-the-art models on multiple datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

“…So, when applying a general pruning technique to VGG16, the performance degradation is negligible in general. Fine-tuning may result in a higher performance of the pruned network Li et al [20] VID [34] GAL [38] than a given pretrained network. In fact, we can observe such a phenomenon in GAL [38] and GBN [31].…”

Section: Pruning Performance Evaluationmentioning

confidence: 99%

Filter Pruning and Re-Initialization via Latent Space Clustering

Lee

Heo

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Authors show that many redundant filters -and feature maps -can be dropped, especially in the case of transfer learning. As already mentioned, [48] rely on a variational approach to sparsify convolutional filters through batch normalization layers. Authors redefine the batch normalization in order to make affine transformation mostly dependant on one single scalar parameter (γ) per filter.…”

Section: Pruning and Compressionmentioning

confidence: 99%

“…Reducing the number of parameters in neural networks can be naturally achieved by low-rank compression [21], [8], [16] or pruning strategies [23], [19], [13]. More recently in [48], a variational approach is developed to sparsify convolutional filters through batch normalization layers. In [35] authors learn separable weights for each filter and bias in convolutional layers and further refine them in a meta learning procedure.…”

Section: Introductionmentioning

confidence: 99%

Transfer Learning by Weighting Convolution

Ayache

Sicre

Artières

2020

2020 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Transferring pretrained deep architectures to datasets with few labels is still a challenge in many real-world situations. This paper presents a new framework to understand convolutional neural networks, by establishing connections between Kronecker factorization and convolutional layers. We then introduce Convolution Weighting Layers that learn a vector of weights for each channel, allowing efficient transfer learning in small training settings, as well as enabling pruning the transferred models. Experiments are conducted on two main settings with few labeled data: transfer learning for classification and transfer learning for retrieval. Two well known convolutional architectures are evaluated on five public datasets. We show that weighting convolutions is efficient to adapt pretrained models to new tasks and that pruned networks conserve good performance.

show abstract

“…Deep CNN also known to solve computer vision issues successfully such as object recognition, semantic segmentation, object detection and video analysis [15]. It is widely used and applied on such as heartbeats classification [16], road crack detection [17], segmentation of blood vessels in retina image, skin cancer and lung lesion [18].…”

Section: Introductionmentioning

confidence: 99%

Comparison Performance of Lymphocyte Classification for Various Datasets using Deep Learning

Safuan¹,

Razali²,

Nurshazwani³

et al. 2020

IJACSA

View full text Add to dashboard Cite

Analyzing and classifying five types of Lymphocyte White Blood Cell (WBC) is important to monitor the lack or excessive amount of cell in human body. These harmful amount of cell must be detected early for the early treatment can be run to the patient. However, the process may be tedious and time consuming as it is done manually by the experts. Other than that, it may yield inaccurate result as it depends on the pathologist skill and experience. This work presents a way that can be the second opinion to the experts using computer aided system as a solution. Convolutional Neural Network (CNN) is applied to the system to avoid complex structure and to eliminate the features extraction process. Three CNN models of mobilenet, resnet and VGG-16 is experimented on three different datasets which are kaggle, LISC and IDB-2. Kaggle, LISC and IDB-2 dataset consist of 6000, 242 and 260 images respectively. The result is divided into two parts which are dataset and model. As for IDB-2 dataset, the best model is VGG with training and validation accuracy of 0.9721 and 0.7913 respectively. While for kaggle and LISC dataset, the best model is resnet as it achieved training accuracy of 0.9713 and 0.9771 respectively. The highest validation accuracy for kaggle is 0.5955 and 0.5781 for LISC. Lastly, the best database that is most suitable for all model is IDB-2 database. It obtained highest training and validation accuracy for all model of mobilenet, resnet and VGG-16.

show abstract

Variational Convolutional Neural Network Pruning

Cited by 256 publications

References 17 publications

Filter Pruning and Re-Initialization via Latent Space Clustering

Filter Pruning and Re-Initialization via Latent Space Clustering

Transfer Learning by Weighting Convolution

Comparison Performance of Lymphocyte Classification for Various Datasets using Deep Learning

Contact Info

Product

Resources

About