Which Metrics for Network Pruning: Final Accuracy? Or Accuracy Drop?

Abstract

Network pruning enables the utilization of deep neural networks in low-resource environments by removing redundant elements in a pre-trained network. To appraise each pruning method, two evaluation metrics are generally adopted, i.e., final accuracy and accuracy drop. Final accuracy represents the ultimate performance of the pruned sub-network after the pruning completes. On the other hand, accuracy drop, a more traditional way, measures the accuracy difference between the baseline model and the final pruned model. In this work, we present several surprising observations which reveal the unfairness of both metrics when assessing the efficacy of pruning approaches. Depending on the choice of baseline network, the value of each pruning method may be completely changed. Specifically, a lower baseline tends to be advantageous for the accuracy drop, whereas a higher baseline usually yields a higher final accuracy. Moreover, to reduce the undesirable dependency on the baseline network, we propose a new reliable averaging method Average from Scratches which uses multiple distinct baselines rather than using a single baseline. Our various investigations point to the necessity for a more thorough analysis on network pruning metrics.

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

“…To address this problem, one possible solution is to adopt more informative metrics (e.g., based on validation loss) as the indicator to the model accuracy. Although some existing schemes [37,51] provided methods to approximately calculate such validation accuracy and loss, they are too computationally expensive for runtime use. Exploring more computationally efficient solutions will be our future work.…”

Is My Pruned Model Trustworthy? PE-Score: A New CAM-Based Evaluation Metric

“…On the other hand, performance can be evaluated in terms of accuracy (or a similar metric, such as the F1-score) or loss (such as cross-entropy). The difference between the original model and the pruned model is then calculated (e.g., accuracy drop [11]), and, usually, the model with the smallest drop in accuracy and the highest number of FLOPs and/or the lowest number of parameters is selected.…”

“…Accuracy (i.e., final accuracy, accuracy drop) and other similar metrics, such as the F1-score, are often used to evaluate the performance. The model size is measured as a function of the number of parameters, while the computational cost is determined by the number of FLOPs (floating point operations) [11][12][13][14]. In this way, the reduction results of the pruned model with respect to the unpruned model can be quantified, taking into account that the smaller the reduction in accuracy (or the smaller the increase in error) and the greater the reduction in both parameters and FLOPs, the better the pruned model.…”

Magnitude and Similarity Based Variable Rate Filter Pruning for Efficient Convolution Neural Networks

“…Given the target pruning rate, average top-1 accuracy along with standard deviation from three experiments on the CIFAR-10 dataset is used to evaluate the capability of the pruning. As suggested by Joo et al [46], we used both final accuracy and accuracy drop w.r.t. baseline accuracy for performance evaluation.…”

“…baseline accuracy for performance evaluation. In addition, we used average from scratches [46] with distinct baselines rather than using a single baseline while averaging the results from different experiments. Again, on the ImageNet dataset, both top-1 and top-5 accuracy are used to evaluate the performance.…”