Cooperative Initialization based Deep Neural Network Training

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

Verma

Rai

et al. 2020

We present a filter correlation based model compression approach for deep convolutional neural networks. Our approach iteratively identifies pairs of filters with the largest pairwise correlations and drops one of the filters from each such pair. However, instead of discarding one of the filters from each such pair naïvely, the model is re-optimized to make the filters in these pairs maximally correlated, so that discarding one of the filters from the pair results in minimal information loss. Moreover, after discarding the filters in each round, we further finetune the model to recover from the potential small loss incurred by the compression. We evaluate our proposed approach using a comprehensive set of experiments and ablation studies. Our compression method yields state-of-the-art FLOPs compression rates on various benchmarks, such as LeNet-5, VGG-16, and ResNet-50,56, while still achieving excellent predictive performance for tasks such as object detection on benchmark datasets.

Section: Introductionmentioning

confidence: 99%

Leveraging Filter Correlations for Deep Model Compression

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

Verma

Rai

et al. 2020

“…Convolutional neural networks (CNNs) have surpassed many traditional machine learning approaches in solving several computer vision tasks such as classification [9,18], segmentation [2], detection [16,12] and others. Various works [24,25,20,23,21,19,13,26,22] have been proposed for efficient deep learning. Researchers have recently been trying to improve CNN performance, by promoting channels (feature maps) that are more relevant [5].…”

Section: Introductionmentioning

confidence: 99%

Accuracy Booster: Performance Boosting using Feature Map Re-calibration

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

Mazumder

Namboodiri

2020

Self Cite

Convolution Neural Networks (CNN) have been extremely successful in solving intensive computer vision tasks. The convolutional filters used in CNNs have played a major role in this success, by extracting useful features from the inputs. Recently researchers have tried to boost the performance of CNNs by re-calibrating the feature maps produced by these filters, e.g., Squeeze-and-Excitation Networks (SENets). These approaches have achieved better performance by Exciting up the important channels or feature maps while diminishing the rest. However, in the process, architectural complexity has increased. We propose an architectural block that introduces much lower complexity than the existing methods of CNN performance boosting while performing significantly better than them. We carry out experiments on the CIFAR, ImageNet and MS-COCO datasets, and show that the proposed block can challenge the state-of-the-art results. Our method boosts the ResNet-50 architecture to perform comparably to the ResNet-152 architecture, which is a three times deeper network, on classification. We also show experimentally that our method is not limited to classification but also generalizes well to other tasks such as object detection.

“…This had led to considerable interest in making the model more efficient, in terms of storage as well as computation [7,15,50,25,47,45,33,52,13,49,48,59]. A popular approach to increase the efficiency of the model is via model compression.…”

Section: Introductionmentioning

confidence: 99%

A "Network Pruning Network" Approach to Deep Model Compression

Verma

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

Namboodiri

et al. 2020

We present a filter pruning approach for deep model compression, using a multitask network. Our approach is based on learning a a pruner network to prune a pre-trained target network. The pruner is essentially a multitask deep neural network with binary outputs that help identify the filters from each layer of the original network that do not have any significant contribution to the model and can therefore be pruned. The pruner network has the same architecture as the original network except that it has a multitask/multi-output last layer containing binary-valued outputs (one per filter), which indicate which filters have to be pruned. The pruner's goal is to minimize the number of filters from the original network by assigning zero weights to the corresponding output feature-maps. In contrast to most of the existing methods, instead of relying on iterative pruning, our approach can prune the network (original network) in one go and, moreover, does not require specifying the degree of pruning for each layer (and can learn it instead). The compressed model produced by our approach is generic and does not need any special hardware/software support. Moreover, augmenting with other methods such as knowledge distillation, quantization, and connection pruning can increase the degree of compression for the proposed approach. We show the efficacy of our proposed approach for classification and object detection tasks.