Dynamic Probabilistic Pruning: A general framework for hardware-constrained pruning at different granularities

Abstract

Unstructured neural network pruning algorithms have achieved impressive compression rates. However, the resulting -typically irregular -sparse matrices hamper efficient hardware implementations, leading to additional memory usage and complex control logic that diminishes the benefits of unstructured pruning. This has spurred structured coarse-grained pruning solutions that prune entire filters or even layers, enabling efficient implementation at the expense of reduced flexibility. Here we propose a flexible new pruning mechanism that facilitates pruning at different granularities (weights, kernels, filters/feature maps), while retaining efficient memory-organization (e.g. pruning exactly k-out-of-n weights for every output neuron, or pruning exactly k-out-of-n kernels for every feature map). We refer to this algorithm as Dynamic Probabilistic Pruning (DPP). DPP leverages the Gumbel-softmax relaxation for differentiable k-out-of-n sampling, facilitating end-to-end optimization. We show that DPP achieves competitive compression rates and classification accuracy when pruning common deep learning models trained on different benchmark datasets for image classification. Relevantly, the non-magnitude-based nature of DPP allows for joint optimization of pruning and weight quantization in order to even further compress the network, which we show as well. Finally, we propose novel information theoretic metrics that show the confidence and pruning diversity of pruning masks within a layer.

A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning

“…Most general, hyper-parameters have been learned -rather than set heuristically -to alleviate the burden of tuning [79], [80], [81]. In case such hyper-parameters relate to the size of the model (e.g., width or depth), NAS (often called pruning in this case) is generally used with the aim to find small models that facilitate implementation in (dedicated) hardware [82], [83], [84], [85], [86], [87], [88], [89]. Moreover, models have been proposed in which the functionality is conditioned upon incoming data.…”

A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning

“…In case such hyper-parameters relate to the size of the model (e.g. width or depth), NAS (often called pruning in this case) is generally used with the aim to find small models that facilitate implementation in (dedicated) hardware [79], [80], [81], [82], [83], [84], [85], [86]. Moreover, models have been proposed in which the functionality is conditioned upon incoming data.…”