Compressing Convolutional Neural Networks by L0 Regularization

Formanek, András; Hadházi, Dániel

doi:10.1109/iccairo47923.2019.00032

Cited by 4 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Score comparison can be performed locally [23], i.e., among the interconnections of the same DNN layer, or globally [10], i.e., among the interconnections of the whole DNN model. As a second criterion, a regularization term [24]- [26] can be imposed on the loss function during training to promote the sparsity of the structure.…”

Section: Pruning Techniques Classificationmentioning

confidence: 99%

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

Prono,

Bich,

Boretti

et al. 2024

Preprint

View full text Add to dashboard Cite

The growing interest in Internet of Things and mobile Artificial Intelligence applications is pushing the investigation on Deep Neural Networks (DNNs) that can operate at the edge using low-resources/energy devices. To obtain such a goal, several pruning techniques have been proposed in the literature. They aim to reduce the number of interconnections - and consequently the size, and the corresponding computing and storage requirements - of DNNs that traditionally rely on classic Multiply-and-ACcumulate (MAC) neurons. In this work, we propose a novel neuron structure based on a Multiply-And-Max/min (MAM) map-reduce paradigm, and we show that by exploiting this new paradigm it is possible to build naturally and aggressively prunable DNN layers, with a negligible loss in performance. This novel structure allows a greater interconnection sparsity when compared to classic MAC-based DNN layers. Moreover, most of the already existing state-of-the-art pruning techniques can be used with MAM layers with little to no changes. To test the pruning performance of MAM, we employ different models - AlexNet, VGG-16 and the more recent ViT-B/16 - and different computer vision datasets - CIFAR-10, CIFAR-100 and ImageNet-1K. Multiple pruning approaches are applied, ranging from single-shot methods to training-dependent and iterative techniques. As a notable example, we test MAM on the ViT-B/16 model fine-tuned on the ImageNet-1K task and apply one-shot gradient-based pruning. We remove interconnections until the model experiences a 3% decrease in accuracy. While MAC-based layers need at least 56.16% remaining interconnections, MAM-based layers achieve the same accuracy with only 0.04%.

show abstract

Section: Pruning Techniques Classificationmentioning

confidence: 99%

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

Prono,

Bich,

Boretti

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Score comparison can be performed locally [20], i.e., among the interconnections of the same DNN layer, or globally [9], i.e., among the interconnections of the whole DNN model. As a second criterion, a regularization term [21], [22] can be imposed to the loss function while training in order to promote the sparsity of the structure.…”

Section: Pruning Techniques Classificationmentioning

confidence: 99%

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

Prono¹,

Bich²,

Mangia³

et al. 2023

Preprint

View full text Add to dashboard Cite

The growing interest in Internet of Things and mobile Artificial Intelligence applications is pushing the investigation on Deep Neural Networks (DNNs) that can operate at the edge using low-resources/energy devices. To obtain such a goal, several pruning techniques have been proposed in the literature. They aim to reduce the number of interconnections -- and consequently the size, and the corresponding computing and storage requirements -- of a DNN relying on classic Multiply-and-ACcumulate (MAC) neurons. In this work, we propose a novel neurons structure based on a Multiply-And-Max/min (MAM) map-reduce paradigm, and we show that by exploiting this new paradigm it is possible to build naturally and aggressively prunable DNN layers, with a negligible loss in performance. In fact, this novel structure allows a greater interconnection sparsity when compared to classic MAC based DNN layers. Moreover, most of the already existing state-of-the-art pruning techniques can be used with MAM layers with little to no changes. As an example, by applying one-shot pruning to a VGG-16 model trained on the ImageNet task, fully connected MAM-based layers need only 0.04% of the total number of interconnections while MAC-based layers need at least 4.33%, with a Top-1 accuracy loss of 3% compared to the maximum achieved accuracy. Additionally, we test Lottery Ticket iterative pruning on AlexNet with CIFAR-100 task. With 0.02% remaining interconnections, the MAC-based model requires 10 training iterations to reach 85% Top-5 accuracy, against 6 iterations with MAM.

show abstract