Filter Distillation for Network Compression

Suau, Xavier; Zappella, uca; Apostoloff, Nicholas

doi:10.1109/wacv45572.2020.9093546

Cited by 34 publications

(26 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Static sparse training with random pruning samples masks within each layer in a random fashion based on pre-defined layer-wise sparsities. The most naive approach is pruning each layer uniformly with the same pruning ratio, i.e., uniform pruning (Mariet & Sra, 2016;He et al, 2017;Suau et al, 2019;Gale et al, 2019). Mocanu et al (2016) proposed a non-uniform and scale-free topology, showing better performance than the dense counterpart when applied to restricted Boltzmann machines (RBMs).…”

Section: Static Sparse Trainingmentioning

confidence: 99%

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

Liu¹,

Chen²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training. In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks. Without any delicate pruning criteria or carefully pursued sparsity structures, we empirically demonstrate that sparsely training a randomly pruned network from scratch can match the performance of its dense equivalent. There are two key factors that contribute to this revival: (i) the network sizes matter: as the original dense networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios; (ii) appropriate layer-wise sparsity ratios can be pre-chosen for sparse training, which shows to be another important performance booster. Simple as it looks, a randomly pruned subnetwork of Wide ResNet-50 can be sparsely trained to outperforming a dense Wide ResNet-50, on ImageNet. We also observed such randomly pruned networks outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness. Overall, our results strongly suggest there is larger-thanexpected room for sparse training at scale, and the benefits of sparsity might be more universal beyond carefully designed pruning. Our source code can be found at https://github.com/VITA-Group/Random_Pruning.

show abstract

Section: Static Sparse Trainingmentioning

confidence: 99%

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

Liu¹,

Chen²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Filter decomposition approaches decompose network matrices into several bases for vector spaces to estimate the informative parameters of the DNNs with low-rank approximation/factorization, thus reducing computation cost of the network [25] such as SVD [5], CP decomposition [21], Tucker decomposition [19], and others, [18] suggests methods to approximate convolutional operations by representing the weight matrix as smaller bases set of 2D separable filters without changing the original number of filters. In [40], Principal Component Analysis (PCA) was applied on max-pooled and flattened feature maps, to compute the amount of information to be preserved in each layer among all layers, enabling integration with each other.…”

Section: Related Workmentioning

confidence: 99%

Toward Compact Deep Neural Networks via Energy-Aware Pruning

Yeom¹,

Shim²,

Hwang³

2021

Preprint

View full text Add to dashboard Cite

Despite of the remarkable performance, modern deep neural networks are inevitably accompanied with a significant amount of computational cost for learning and deployment, which may be incompatible with their usage on edge devices. Recent efforts to reduce these overheads involves pruning and decomposing the parameters of various layers without performance deterioration. Inspired by several decomposition studies, in this paper, we propose a novel energy-aware pruning method that quantifies the importance of each filter in the network using nuclear-norm (NN). Proposed energy-aware pruning leads to state-of-the art performance for Top-1 accuracy, FLOPs, and parameter reduction across a wide range of scenarios with multiple network architectures on CIFAR-10 and ImageNet after fine-grained classification tasks. On toy experiment, despite of no fine-tuning, we can visually observe that NN not only has little change in decision boundaries across classes, but also clearly outperforms previous popular criteria. We achieve competitive results with 40.4/49.8% of FLOPs and 45.9/52.9% of parameter reduction with 94.13/94.61% in the Top-1 accuracy with ResNet-56/110 on CIFAR-10, respectively. In addition, our observations are consistent for a variety of different pruning setting in terms of data size as well as data quality which can be emphasized in the stability of the acceleration and compression with negligible accuracy loss. Our code is available at https:// github.com/ nota-github/ nota-pruning-rank.

show abstract

“…Structured pruning, as considered in this work, prunes parameter groups instead of individual weights, allowing speedups to be achieved without sparse computation (Li et al, 2016). Many empirical heuristics for structured pruning exist; e.g., pruning parameter groups with low 1 norm (Li et al, 2016;Liu et al, 2017), measuring the gradient-based sensitivity of parameter groups , preserving network output (He et al, 2017;Luo et al, 2017;Yu et al, 2017), and more (Suau et al, 2020;Chin et al, 2019;Huang and Wang, 2018;Molchanov et al, 2016). Pruning typically follows a three-step process of pre-training, pruning, and fine-tuning (Li et al, 2016;.…”

Section: Related Workmentioning

confidence: 99%

i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery

Wolfe¹,

Kyrillidis²

2021

Preprint

View full text Add to dashboard Cite

We propose a novel, structured pruning algorithm for neural networks-the iterative, Sparse Structured Pruning algorithm, dubbed as i-SpaSP. Inspired by ideas from sparse signal recovery, i-SpaSP operates by iteratively identifying a larger set of important parameter groups (e.g., filters or neurons) within a network that contribute most to the residual between pruned and dense network output, then thresholding these groups based on a smaller, pre-defined pruning ratio. For both two-layer and multi-layer network architectures with ReLU activations, we show the error induced by pruning with i-SpaSP decays polynomially, where the degree of this polynomial becomes arbitrarily large based on the sparsity of the dense network's hidden representations. In our experiments, i-SpaSP is evaluated across a variety of datasets (i.e., MNIST and ImageNet) and architectures (i.e., feed forward networks, ResNet34, and MobileNetV2), where it is shown to discover high-performing sub-networks and improve upon the pruning efficiency of provable baseline methodologies by several orders of magnitude. Put simply, i-SpaSP is easy to implement with automatic differentiation, achieves strong empirical results, comes with theoretical convergence guarantees, and is efficient, thus distinguishing itself as one of the few computationally efficient, practical, and provable pruning algorithms.

show abstract

Filter Distillation for Network Compression

Cited by 34 publications

References 80 publications

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

Toward Compact Deep Neural Networks via Energy-Aware Pruning

i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery

Contact Info

Product

Resources

About