Recent Advances on Neural Network Pruning at Initialization

Abstract

In recent years, spiking neural networks (SNNs) have received extensive attention in brain-inspired intelligence due to their rich spatially-temporal dynamics, various encoding methods, and event-driven characteristics that naturally fit the neuromorphic hardware. With the development of SNNs, brain-inspired intelligence, an emerging research field inspired by brain science achievements and aiming at artificial general intelligence, is becoming hot. This paper reviews recent advances and discusses new frontiers in SNNs from five major research topics, including essential elements (i.e., spiking neuron models, encoding methods, and topology structures), neuromorphic datasets, optimization algorithms, software, and hardware frameworks. We hope our survey can help researchers understand SNNs better and inspire new works to advance this field.

One-shot Network Pruning at Initialization with Discriminative Image Patches

“…Neural network pruning research started in the late 1980s [31], and it is become increasingly popular until recently [16,38]. There are three steps in a typical pruning pipeline:…”

“…Pruning has been regarded as a post-processing approach to solving the problem of redundancy [3,6,7,13,19,23,34]. However, a novel pruning paradigm, Network Pruning at Initialization (PaI), is out of this pipeline [38], which trains a sparse subnetwork from a randomly initialized dense network instead of pruning a pre-trained network. PaI seeks to achieve full accuracy at lower training costs.…”

“…One of the notable PaI efforts is the lottery ticket hypothesis (LTH) [11], which selects subnetwork masks from pre-trained dense networks. This outstanding work sparks an interest in the deep learning community, and numerous related PaI researches have emerged [4,25,27,38,41,43]. Most traditional PaI methods, such as LTH, dramatically raise the pruning cost since iterative pruning is needed.…”

“…Such pruned neural networks can significantly decrease the computational demands of inference by using specific methods [8,14]. Among the various pruning methods developed so far, Network Pruning at Initialization (PaI) has attracted considerable attention since it provides a possibility to train sparse networks at lower costs [38]. Specifically, PaI aims to achieve (close to) full accuracy (the accuracy reached by the dense network) by training a sparse subnetwork from a randomly initialized dense network.…”

“…In PaI research, the pruning criterion is the key focus [11,21,26,28,29,35,38]. Most PaI works involve iterative pruning processing to improve performance while drastically increasing training costs.…”