Interpretations Steered Network Pruning via Amortized Inferred Saliency Maps

Ganjdanesh, Alireza; Gao, Shangqian; Huang, Heng

doi:10.1007/978-3-031-19803-8_17

Cited by 10 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A recently proposed method selects useful deep features from a discriminative dimension reduction perspective, through Fisher Linear Discriminant Analysis so that the method captures both the final separation of classes and their holistic inter-layer dependence [25]. The use of Amortized Explanation Models (AEMs) that utilize information from both inputs and outputs have also been proposed in order to predict in real-time smooth saliency masks and leverage the interpretations of the model to steer the pruning process [26]. It has also been proposed to evaluate gradient-based salience to measure the importance of a channel and to prune the lowest-scoring channels and their corresponding filters according to the pruning rate [27].…”

Section: Pruning Methods and Pruning Evaluationmentioning

confidence: 99%

Sp2PS: Pruning Score by Spectral and Spatial Evaluation of CAM Images

Renza,

Ballesteros

2023

Informatics

View full text Add to dashboard Cite

CNN models can have millions of parameters, which makes them unattractive for some applications that require fast inference times or small memory footprints. To overcome this problem, one alternative is to identify and remove weights that have a small impact on the loss function of the algorithm, which is known as pruning. Typically, pruning methods are compared in terms of performance (e.g., accuracy), model size and inference speed. However, it is unusual to evaluate whether a pruned model preserves regions of importance in an image when performing inference. Consequently, we propose a metric to assess the impact of a pruning method based on images obtained by model interpretation (specifically, class activation maps). These images are spatially and spectrally compared and integrated by the harmonic mean for all samples in the test dataset. The results show that although the accuracy in a pruned model may remain relatively constant, the areas of attention for decision making are not necessarily preserved. Furthermore, the performance of pruning methods can be easily compared as a function of the proposed metric.

show abstract

Section: Pruning Methods and Pruning Evaluationmentioning

confidence: 99%

Sp2PS: Pruning Score by Spectral and Spatial Evaluation of CAM Images

Renza,

Ballesteros

2023

Informatics

View full text Add to dashboard Cite

show abstract

“…Network Pruning: Model compression (Ghimire, Kil, and Kim 2022) is a well-studied topic, and proposed methods have primarily focused on compressing CNN classifiers. They use various techniques like weight pruning (Han et al 2015), light architecture design (Tan and Le 2019;Howard et al 2017), weight quantization (Rastegari et al 2016), structural pruning (Li et al 2017;Ye et al 2020;Ganjdanesh, Gao, and Huang 2022;He et al 2018), knowledge distillation (Ba and Caruana 2014), and NAS (Wu et al 2019;Ganjdanesh, Gao, and Huang 2023). We focus on pruning GANs, which is a more challenging task due to the instability of their training (Wang et al 2020).…”

Section: Related Workmentioning

confidence: 99%

Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold

Ganjdanesh,

Gao,

Alipanah

et al. 2024

AAAI

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have shown remarkable success in modeling complex data distributions for image-to-image translation. Still, their high computational demands prohibit their deployment in practical scenarios like edge devices. Existing GAN compression methods mainly rely on knowledge distillation or convolutional classifiers' pruning techniques. Thus, they neglect the critical characteristic of GANs: their local density structure over their learned manifold. Accordingly, we approach GAN compression from a new perspective by explicitly encouraging the pruned model to preserve the density structure of the original parameter-heavy model on its learned manifold. We facilitate this objective for the pruned model by partitioning the learned manifold of the original generator into local neighborhoods around its generated samples. Then, we propose a novel pruning objective to regularize the pruned model to preserve the local density structure over each neighborhood, resembling the kernel density estimation method. Also, we develop a collaborative pruning scheme in which the discriminator and generator are pruned by two pruning agents. We design the agents to capture interactions between the generator and discriminator by exchanging their peer's feedback when determining corresponding models' architectures. Thanks to such a design, our pruning method can efficiently find performant sub-networks and can maintain the balance between the generator and discriminator more effectively compared to baselines during pruning, thereby showing more stable pruning dynamics. Our experiments on image translation GAN models, Pix2Pix and CycleGAN, with various benchmark datasets and architectures demonstrate our method's effectiveness.

show abstract

“…The regularization in Eq. 4 has also been utilized for network pruning (Ganjdanesh, Gao, and Huang 2022;Gao et al 2020). As the round(•) function is not differentiable, we calculate the gradients of L param w.r.t θ sp using the Straight Through Estimator (Bengio, Léonard, and Courville 2013).…”

Section: Size Predictormentioning

confidence: 99%

EffConv: Efficient Learning of Kernel Sizes for Convolution Layers of CNNs

Ganjdanesh

Gao

Huang

2023

AAAI

View full text Add to dashboard Cite

Determining kernel sizes of a CNN model is a crucial and non-trivial design choice and significantly impacts its performance. The majority of kernel size design methods rely on complex heuristic tricks or leverage neural architecture search that requires extreme computational resources. Thus, learning kernel sizes, using methods such as modeling kernels as a combination of basis functions, jointly with the model weights has been proposed as a workaround. However, previous methods cannot achieve satisfactory results or are inefficient for large-scale datasets. To fill this gap, we design a novel efficient kernel size learning method in which a size predictor model learns to predict optimal kernel sizes for a classifier given a desired number of parameters. It does so in collaboration with a kernel predictor model that predicts the weights of the kernels - given kernel sizes predicted by the size predictor - to minimize the training objective, and both models are trained end-to-end. Our method only needs a small fraction of the training epochs of the original CNN to train these two models and find proper kernel sizes for it. Thus, it offers an efficient and effective solution for the kernel size learning problem. Our extensive experiments on MNIST, CIFAR-10, STL-10, and ImageNet-32 demonstrate that our method can achieve the best training time vs. accuracy trade-off compared to previous kernel size learning methods and significantly outperform them on challenging datasets such as STL-10 and ImageNet-32. Our implementations are available at https://github.com/Alii-Ganjj/EffConv.

show abstract

Interpretations Steered Network Pruning via Amortized Inferred Saliency Maps

Cited by 10 publications

References 34 publications

Sp2PS: Pruning Score by Spectral and Spatial Evaluation of CAM Images

Sp2PS: Pruning Score by Spectral and Spatial Evaluation of CAM Images

Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold

EffConv: Efficient Learning of Kernel Sizes for Convolution Layers of CNNs

Contact Info

Product

Resources

About