CUP: Cluster Pruning for Compressing Deep Neural Networks

Abstract

We propose Cluster Pruning (CUP) for compressing and accelerating deep neural networks. Our approach prunes similar filters by clustering them based on features derived from both the incoming and outgoing weight connections. With CUP, we overcome two limitations of prior work-(1) nonuniform pruning: CUP can efficiently determine the ideal number of filters to prune in each layer of a neural network. This is in contrast to prior methods that either prune all layers uniformly or otherwise use resource-intensive methods such as manual sensitivity analysis or reinforcement learning to determine the ideal number. (2) Single-shot operation:We extend CUP to CUP-SS (for CUP single shot) whereby pruning is integrated into the initial training phase itself. This leads to large savings in training time compared to traditional pruning pipelines. Through extensive evaluation on multiple datasets (MNIST, CIFAR-10, and Imagenet) and models (VGG-16, Resnets-18/34/56) we show that CUP outperforms recent state of the art. Specifically, CUP-SS achieves 2.2× flops reduction for a Resnet-50 model trained on Imagenet while staying within 0.9% top-5 accuracy. It saves over 14 hours in training time with respect to the original Resnet-50. Code to reproduce results is available here 1 .

NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks

“…Olah et al [40] highlights many examples of similar neurons in InceptionV1 and visualizes which concepts are detected by such neurons; however, the examples are manually curated by the authors. Identifying neurons that discover similar concepts also has practical benefits: in the neural network compression community, several methods [14,15,21,25,56] leverage potential neuron redundancies to generate compressed models while maintaining prediction accuracy. Even though these methods can measure neurons' similarity, there is limited work in interpreting their semantic similarity.…”

“…Existing research on DNN interpretability tends to focus on inspecting individual neurons [23,37,42]. While helpful, neuron-level inspection cannot easily reveal how clusters of neurons may detect the same concept, even though it is common for multiple neurons to detect similar features [15,21,25,56]. As a result, users can easily miss higher-order interactions that explain how DNNs operate.…”

“…As DNNs are increasingly used in an everincreasing variety of applications, our approaches can help practitioners and researchers assess the effectiveness of their ideas. For example, in the neural network compression community, several methods [15,21,25,56] leverage potential neuron redundancies to generate compressed models while maintaining prediction accuracy. NEURO-CARTOGRAPHY can help researchers interpret the semantic similarity between the compressed model and the original, uncompressed models, which helps them assess if their techniques are indeed preserving the "gist" of the knowledge important for prediction, or if they are leveraging some other features of the data of the model.…”

Convolutional Neural Network Pruning with Structural Redundancy Reduction

“…(3) Similarity measurement. These approaches use various strategies, such as geometric median [15] and clustering [51,6], to identify the most replaceable filters, or those functionally share the most similarity with others.…”

“…We compare the performance of our approach with several recent channel pruning methods, namely, minimum weight (MW) [24], Taylor expansion [35], average percentage of zero activation neurons (APoZ) [17], soft filter pruning (SFP) [14], discrimination-aware channel pruning (DCP) [52], neuron importance score propagation (NISP) [47], slimmable neural networks (SNN) [46], autopruner (AP) [30], generative adversarial learning (GAL) [27], geometric median (GM) [15], transformable architecture search (TAS) [5], cluster pruning (CUP) [6], ABC [26], trained rank pruning (TRP) [45], soft channel pruning (SCP) [19], and high-hank (HRank) [25].…”