Activation Density driven Energy-Efficient Pruning in Training

Abstract

The process of neural network pruning with suitable fine-tuning and retraining can yield networks with considerably fewer parameters than the original with comparable degrees of accuracy. Typically, pruning methods require large, pre-trained networks as a starting point from which they perform a timeintensive iterative pruning and retraining algorithm. We propose a novel pruning in-training method that prunes a network realtime during training, reducing the overall training time to achieve an optimal compressed network. To do so, we introduce an activation density based analysis that identifies the optimal relative sizing or compression for each layer of the network. Our method removes the need for pre-training and is architecture agnostic, allowing it to be employed on a wide variety of systems. For VGG-19 and ResNet18 on CIFAR-10, CIFAR-100, and TinyImageNet, we obtain exceedingly sparse networks (up to 200× reduction in parameters and > 60× reduction in inference compute operations in the best case) with comparable accuracies (up to 2%-3% loss with respect to the baseline network). By reducing the network size periodically during training, we achieve total training times that are shorter than those of previously proposed pruning methods. Furthermore, training compressed networks at different epochs with our proposed method yields considerable reduction in training compute complexity (1.6 × −3.2× lower) at near iso-accuracy as compared to a baseline network trained entirely from scratch.

Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks

“…There is relative lack of research on networks that have both forms of sparsity (exceptions are [1,94]). In some scenarios networks trained without explicit activation sparsity end up with highly sparse activations anyway [8,19,20,33]. This is encouraging because it suggests that sparse activations may naturally be an optimal outcome.…”

“…In our implementation we use k-WTA to achieve activation sparsity. Another approach is to remove entire channels in convolutional layers during training through a structured pruning process [19,44,73]. In [19] they notice that activations naturally become sparse during training and use a measure of sparsity to gradually prune channels.…”

“…Another approach is to remove entire channels in convolutional layers during training through a structured pruning process [19,44,73]. In [19] they notice that activations naturally become sparse during training and use a measure of sparsity to gradually prune channels. In [73] the method is extended to incorporate mixed precision quantization based on activation sparsity and then evaluated on a hardware simulation platform.…”

“…As a result in principle all phases of training should see significant speedup through Complementary Sparsity. Structured pruning steps [19,44,73], as discussed earlier, can also be folded in to further accelerate training. These techniques may require highly flexible circuitry, and as such, FPGAs may serve as a better platform than GPUs for developing these ideas.…”