pSConv: A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

Kundu, Souvik; Prakash, Saurav; Akrami, Haleh; Beerel, Peter A.; Chugg, K.M.

doi:10.1109/allerton.2019.8919683

Cited by 9 publications

(6 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our model can be extended to fullyconnected layers with f l i and f l o as the number of input and output features respectively. In particular, for an ANN, the total number of FLOPS for layer l, denoted F l AN N , is shown in row 1 of Table III [49], [50]. The formula can be easily adjusted for an SNN in which the number of FLOPs at layer l is a function of the average spiking activity at the layer (ζ l ) denoted as F l SN N in Table III.…”

Section: B Reduction In Flops and Compute Energymentioning

confidence: 99%

Training Energy-Efficient Deep Spiking Neural Networks with Single-Spike Hybrid Input Encoding

Datta¹,

Kundu²,

A.³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have emerged as an attractive alternative to traditional deep learning frameworks, since they provide higher computational efficiency in event driven neuromorphic hardware. However, the state-of-the-art (SOTA) SNNs suffer from high inference latency, resulting from inefficient input encoding and training techniques. The most widely used input coding schemes, such as Poisson based rate-coding, do not leverage the temporal learning capabilities of SNNs. This paper presents a training framework for low-latency energyefficient SNNs that uses a hybrid encoding scheme at the input layer in which the analog pixel values of an image are directly applied during the first timestep and a novel variant of spike temporal coding is used during subsequent timesteps. In particular, neurons in every hidden layer are restricted to fire at most once per image which increases activation sparsity. To train these hybrid-encoded SNNs, we propose a variant of the gradient descent based spike timing dependent backpropagation (STDB) mechanism using a novel cross entropy loss function based on both the output neurons' spike time and membrane potential. The resulting SNNs have reduced latency and high activation sparsity, yielding significant improvements in computational efficiency. In particular, we evaluate our proposed training scheme on image classification tasks from CIFAR-10 and CIFAR-100 datasets on several VGG architectures. We achieve top-1 accuracy of 66.46% with 5 timesteps on the CIFAR-100 dataset with ∼125× less compute energy than an equivalent standard ANN. Additionally, our proposed SNN performs 5-300× faster inference compared to other state-of-the-art rate or temporally coded SNN models.

show abstract

Section: B Reduction In Flops and Compute Energymentioning

confidence: 99%

Training Energy-Efficient Deep Spiking Neural Networks with Single-Spike Hybrid Input Encoding

Datta¹,

Kundu²,

A.³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Behavior cloning, a method of teaching a machine learning algorithm how to complete a task, was implemented for the training of this model. This method was chosen because of its ease of implementation and its ability to "yield optimal results efficiently" [15]. Behavior cloning allows for pre-collected data from a human expert to be treated as a ground truth dataset.…”

Section: Trainingmentioning

confidence: 99%

Impact of Model Architecture Against Adversarial Example's Effectivity

Karnala

Campbell

2021

J Stud Res

View full text Add to dashboard Cite

The purpose of this study is to gain an understanding of the impact of model architecture on the efficacy of adversarial examples against machine learning systems implemented in self-driving applications. Prior research shows how to create and train against adversarial examples in many use cases; however, there is no definite understanding of how a machine learning model’s architecture affects the efficacy of adversarial examples. Data was collected through an experimental setting involving end-to-end self-driving models trained through behavioral cloning. Three model types were tested based on popular frameworks for machine learning algorithms dealing with images. Results showed a statistically significant difference in the impact of adversarial examples between these models. This means that certain model types and architectures are more susceptible to attacks. Therefore, the conclusion can be made that model architecture does impact the efficacy of adversarial examples; however, this is potentially limited to closed-loop, end-to-end systems in which algorithms make the entire decision. Future research should investigate what specific structure within models causes increased susceptibility to adversarial attacks.

show abstract

“…This section first describes pSConv, a form of pre-defined sparse kernel based convolution that we initially proposed in [1]. It then describe how we introduce periodicity to this framework to reduce the overhead of managing sparse matrix representations.…”

Section: Pre-defined Sparsitymentioning

confidence: 99%

“…This paper proposes pre-defined sparse convolutions to improve energy and storage efficiency during both training and inference. We refer to this approach as pSConv and presented initial simulation results that show negligible performance degradation compared to fully-connected baseline models in [1]. However, as mentioned earlier, unstructured forms of pSConv may not lead to energy reductions due to the overhead of managing their sparse matrix representations.…”

Section: Introductionmentioning

confidence: 99%

Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks

Kundu

Nazemi

Pedram

et al. 2020

IEEE Trans. Comput.

Self Cite

View full text Add to dashboard Cite

The high energy cost of processing deep convolutional neural networks impedes their ubiquitous deployment in energy-constrained platforms such as embedded systems and IoT devices. This work introduces convolutional layers with pre-defined sparse 2D kernels that have support sets that repeat periodically within and across filters. Due to the efficient storage of our periodic sparse kernels, the parameter savings can translate into considerable improvements in energy efficiency due to reduced DRAM accesses, thus promising significant improvements in the trade-off between energy consumption and accuracy for both training and inference. To evaluate this approach, we performed experiments with two widely accepted datasets, CIFAR-10 and Tiny ImageNet in sparse variants of the ResNet18 and VGG16 architectures. Compared to baseline models, our proposed sparse variants require up to ∼82% fewer model parameters with 5.6× fewer FLOPs with negligible loss in accuracy for ResNet18 on CIFAR-10. For VGG16 trained on Tiny ImageNet, our approach requires 5.8× fewer FLOPs and up to ∼83.3% fewer model parameters with a drop in top-5 (top-1) accuracy of only 1.2% (∼2.1%). We also compared the performance of our proposed architectures with that of ShuffleNet and MobileNetV2. Using similar hyperparameters and FLOPs, our ResNet18 variants yield an average accuracy improvement of ∼2.8%.

show abstract

pSConv: A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

Cited by 9 publications

References 21 publications

Training Energy-Efficient Deep Spiking Neural Networks with Single-Spike Hybrid Input Encoding

Training Energy-Efficient Deep Spiking Neural Networks with Single-Spike Hybrid Input Encoding

Impact of Model Architecture Against Adversarial Example's Effectivity

Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks

Contact Info

Product

Resources

About