Differentiable Mask for Pruning Convolutional and Recurrent Networks

Ramakrishnan, Ramchalam Kinattinkara; Sari, Eyyüb; Nia, Vahid Partovi

doi:10.1109/crv50864.2020.00037

Cited by 6 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the unstructured nature of the removed parameters makes it difficult to operate the network. Recently, several works have been proposed to prune in a structured way where a whole kernel, a filter of even a layer is pruned according to a specific criterion [19][20][21][22].…”

Section: State Of the Art On Embedded Execution Of Quantized Neural N...mentioning

confidence: 99%

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Novac

Hacene

Pégatoquet

et al. 2021

Sensors

View full text Add to dashboard Cite

Embedding Artificial Intelligence onto low-power devices is a challenging task that has been partly overcome with recent advances in machine learning and hardware design. Presently, deep neural networks can be deployed on embedded targets to perform different tasks such as speech recognition, object detection or Human Activity Recognition. However, there is still room for optimization of deep neural networks onto embedded devices. These optimizations mainly address power consumption, memory and real-time constraints, but also an easier deployment at the edge. Moreover, there is still a need for a better understanding of what can be achieved for different use cases. This work focuses on quantization and deployment of deep neural networks onto low-power 32-bit microcontrollers. The quantization methods, relevant in the context of an embedded execution onto a microcontroller, are first outlined. Then, a new framework for end-to-end deep neural networks training, quantization and deployment is presented. This framework, called MicroAI, is designed as an alternative to existing inference engines (TensorFlow Lite for Microcontrollers and STM32Cube.AI). Our framework can indeed be easily adjusted and/or extended for specific use cases. Execution using single precision 32-bit floating-point as well as fixed-point on 8- and 16 bits integers are supported. The proposed quantization method is evaluated with three different datasets (UCI-HAR, Spoken MNIST and GTSRB). Finally, a comparison study between MicroAI and both existing embedded inference engines is provided in terms of memory and power efficiency. On-device evaluation is done using ARM Cortex-M4F-based microcontrollers (Ambiq Apollo3 and STM32L452RE).

show abstract

Section: State Of the Art On Embedded Execution Of Quantized Neural N...mentioning

confidence: 99%

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Novac

Hacene

Pégatoquet

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…In other words, if the scaling factor is 0, it is guaranteed that the corresponding unit has no contribution to recognition. We can include mask based pruning methods [33,57] under this category as well since the basic principle is the same. Although quite useful in practice, this approach does not tell us anything about the nature of a "good unit" for recognition.…”

Section: Related Workmentioning

confidence: 99%

A Deeper Look into Convolutions via Pruning

Cugu¹,

Akbaş²

2021

Preprint

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are able to attain better visual recognition performance than fully connected neural networks despite having much less parameters due to their parameter sharing principle. Hence, modern architectures are designed to contain a very small number of fully-connected layers, often at the end, after multiple layers of convolutions. It is interesting to observe that we can replace large fully-connected layers with relatively small groups of tiny matrices applied on the entire image. Moreover, although this strategy already reduces the number of parameters, most of the convolutions can be eliminated as well, without suffering any loss in recognition performance. However, there is no solid recipe to detect this hidden subset of convolutional neurons that is responsible for the majority of the recognition work. Hence, in this work, we use the matrix characteristics based on eigenvalues in addition to the classical weight-based importance assignment approach for pruning to shed light on the internal mechanisms of a widely used family of CNNs, namely residual neural networks (ResNets), for the image classification problem using CIFAR-10, CIFAR-100 and Tiny ImageNet datasets. The codes are available at https://github.com/cuguilke/psykedelic.

show abstract

“…Yamamoto et al [9] use a channel-pruning technique based on an attention mechanism, where attention blocks are introduced into each layer and updated during training to evaluate the importance of each channel. In [10], the authors propose a learnable differentiable mask, that aims at finding out during training process the less important neurons, channels or even layers and prune them. In [11], the authors propose to give to the DNN the ability to decide during training which criterion should be considered for each layer when pruning.…”

Section: Dnn Compressionmentioning

confidence: 99%

DecisiveNets: Training Deep Associative Memories to Solve Complex Machine Learning Problems

Gripon,

Lassance,

Hacene

2020

Preprint

View full text Add to dashboard Cite

Learning deep representations to solve complex machine learning tasks has become the prominent trend in the past few years. Indeed, Deep Neural Networks are now the golden standard in domains as various as computer vision, natural language processing or even playing combinatorial games. However, problematic limitations are hidden behind this surprising universal capability. Among other things, explainability of the decisions is a major concern, especially since deep neural networks are made up of a very large number of trainable parameters. Moreover, computational complexity can quickly become a problem, especially in contexts constrained by real time or limited resources. Therefore, understanding how information is stored and the impact this storage can have on the system remains a major and open issue. In this chapter, we introduce a method to transform deep neural network models into deep associative memories, with simpler, more explicable and less expensive operations. We show through experiments that these transformations can be done without penalty on predictive performance. The resulting deep associative memories are excellent candidates for artificial intelligence that is easier to theorize and manipulate.

show abstract

Differentiable Mask for Pruning Convolutional and Recurrent Networks

Cited by 6 publications

References 12 publications

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Quantization and Deployment of Deep Neural Networks on Microcontrollers

A Deeper Look into Convolutions via Pruning

DecisiveNets: Training Deep Associative Memories to Solve Complex Machine Learning Problems

Contact Info

Product

Resources

About