A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

Abstract

<p>The growing interest in Internet of Things and mobile Artificial Intelligence applications is pushing the investigation on Deep Neural Networks (DNNs) that can operate at the edge using low-resources/energy devices.</p> <p>To obtain such a goal, several pruning techniques have been proposed in the literature. They aim to reduce the number of interconnections -- and consequently the size, and the corresponding computing and storage requirements -- of a DNN relying on classic Multiply-and-ACcumulate (MAC) neurons.</p> <p>In this work, we propose a novel neurons structure based on a Multiply-And-Max/min (MAM) map-reduce paradigm, and we show that by exploiting this new paradigm it is possible to build naturally and aggressively prunable DNN layers, with a negligible loss in performance. In fact, this novel structure allows a greater interconnection sparsity when compared to classic MAC based DNN layers. Moreover, most of the already existing state-of-the-art pruning techniques can be used with MAM layers with little to no changes. As an example, by applying one-shot pruning to a VGG-16 model trained on the ImageNet task, fully connected MAM-based layers need only 0.04% of the total number of interconnections while MAC-based layers need at least 4.33%, with a Top-1 accuracy loss of 3% compared to the maximum achieved accuracy. Additionally, we test Lottery Ticket iterative pruning on AlexNet with CIFAR-100 task. With 0.02% remaining interconnections, the MAC-based model requires 10 training iterations to reach 85% Top-5 accuracy, against 6 iterations with MAM.</p>