Activation Density Driven Efficient Pruning in Training

Foldy-Porto, Timothy; Venkatesha, Yeshwanth; Panda, Priyadarshini

doi:10.1109/icpr48806.2021.9413182

Cited by 4 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As such, many other structured pruning techniques have been suggested since [3]- [10], [18], [19], [22], [29], [31], [37]- [39]. Studies such as [14], [24], [25] leverage activations to prune the network, as is followed in our work however they all use ℓ-norm based metrics to do so.…”

Section: Related Workmentioning

confidence: 99%

“…Most of these scoring techniques are based on some abstraction of the way DNNs work, for example, a simple one being the smaller the ℓ-norm (or the magnitude) of a parameter the less its assumed importance, as used in studies by [8]- [12], [14], [18]- [23]. Other approaches using activations (or outputs) of neurons as an importance score such as [24]- [26] also use ℓ-norms in some way. As designing DNNs can still be considered a somewhat elusive art, it is difficult to justify these abstractions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Class-Separation Preserving Pruning for Deep Neural Networks

Preet

Boydell

John

2024

IEEE Trans. Artif. Intell.

View full text Add to dashboard Cite

Neural network pruning has been deemed essential in the deployment of deep neural networks on resourceconstrained edge devices, greatly reducing the number of network parameters without drastically compromising accuracy. A class of techniques proposed in the literature assigns an importance score to each parameter and prunes those of the least importance. However, most of these methods are based on generalised estimations of the importance of each parameter, ignoring the context of the specific task at hand. In this paper, we propose a task specific pruning approach, CSPrune, which is based on how efficiently a neuron or a convolutional filter is able to separate classes. Our axiomatic approach assigns an importance score based on how separable different classes are in the output activations or feature maps, preserving separation of classes which avoids reduction in classification accuracy. Additionally, most pruning algorithms prune individual connections or weights leading to a sparse network without taking into account whether the hardware the network is deployed on can take advantage of that sparsity or not. CSPrune prunes whole neurons or filters which results in a more structured pruned network whose sparsity can be more efficiently utilised by the hardware. We evaluate our pruning method against various benchmark datasets, both small and large, and network architectures and show that our approach outperforms comparable pruning techniques.Impact Statement-Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy across a variety of tasks such as classification, semantic segmentation, robot navigation, etc. Typically, these DNNs contain a few hundred thousand to a few million parameters which require a lot of memory, computational resources, and power, which typical devices at the edge are severely constrained in. Several techniques like quantization, tensor decomposition, and pruning have been proposed to compress DNNs for easy deployment on these edge devices. Our novel pruning methodology, CSPrune, proposed in this work compresses the DNNs whilst preserving accuracy. We compare it with several other studies on large datasets and DNNs and show it performs better in many cases.

show abstract