Topological Insights into Sparse Neural Networks

Liu, Shiwei; Lee, Tim van der; Yaman, Anil; Atashgahi, Zahra; Ferraro, Davide; Sokar, Ghada; Pechenizkiy, Mykola; Mocanu, Decebal Constantin

doi:10.48550/arxiv.2006.14085

Cited by 3 publications

(1 citation statement)

References 23 publications

(39 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Works from [5,32] also show that the sparse training achieves better performance than iteratively pruning a pre-trained dense model and static sparse neural networks. Moreover, Liu et al [23] demonstrated that there is a plenitude of sparse sub-networks with very different topologies that achieve the same performance.…”

Section: Spacenet Approach For Continual Learningmentioning

confidence: 99%

SpaceNet: Make Free Space For Continual Learning

Sokar,

Mocanu,

Pechenizkiy

2020

Preprint

Self Cite

View full text Add to dashboard Cite

The continual learning (CL) paradigm aims to enable neural networks to learn tasks continually in a sequential fashion. The fundamental challenge in this learning paradigm is catastrophic forgetting previously learned tasks when the model is optimized for a new task, especially when their data is not accessible. Current architectural-based methods aim at alleviating the catastrophic forgetting problem but at the expense of expanding the capacity of the model. Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e.g. class incremental learning scenario). In this work, we propose a novel architectural-based method referred as SpaceNet for class incremental learning scenario where we utilize the available fixed capacity of the model intelligently. SpaceNet trains sparse deep neural networks from scratch in an adaptive way that compresses the sparse connections of each task in a compact number of neurons. The adaptive training of the sparse connections results in sparse representations that reduce the interference between the tasks. Experimental results show the robustness of our proposed method against catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing the available capacity of the model, leaving space for more tasks to be learned. In particular, when SpaceNet is tested on the well-known benchmarks for CL: split MNIST, split Fashion-MNIST, and CIFAR-10/100, it outperforms regularization-based methods by a big performance gap. Moreover, it achieves better performance than architectural-based methods without model expansion and achieved comparable results with rehearsal-based methods, while offering a huge memory reduction.

show abstract

Section: Spacenet Approach For Continual Learningmentioning

confidence: 99%

SpaceNet: Make Free Space For Continual Learning

Sokar,

Mocanu,

Pechenizkiy

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Selfish Sparse RNN Training

Liu¹,

Mocanu²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

Sparse neural networks have been widely applied to reduce the necessary resource requirements to train and deploy over-parameterized deep neural networks. For inference acceleration, methods that induce sparsity from a pre-trained dense network (dense-to-sparse training) work effectively. Recently, dynamic sparse training (DST) has been proposed to train sparse neural networks without pre-training a dense network (sparse-to-sparse training), so that the training process can also be accelerated. However, previous sparse-to-sparse methods mainly focus on Multilayer Perceptron Networks (MLPs) and Convolutional Neural Networks (CNNs), failing to match the performance of dense-to-sparse methods in Recurrent Neural Networks (RNNs) setting. In this paper, we propose an approach to train sparse RNNs with a fixed parameter count in one single run, without compromising performance. During training, we allow RNN layers to have a non-uniform redistribution across cell gates for a better regularization. Further, we introduce SNT-ASGD, a variant of the averaged stochastic gradient optimizer, which significantly improves the performance of all sparse training methods for RNNs. Using these strategies, we achieve state-of-the-art sparse training results, better than the dense-to-sparse methods, with various types of RNNs on Penn TreeBank and Wikitext-2 datasets.

show abstract

Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch

Zhou¹,

Ma²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. It can be generally categorized into unstructured fine-grained sparsity that zeroes out multiple individual weights distributed across the neural network, and structured coarsegrained sparsity which prunes blocks of sub-networks of a neural network. Finegrained sparsity can achieve a high compression ratio but is not hardware friendly and hence receives limited speed gains. On the other hand, coarse-grained sparsity cannot concurrently achieve both apparent acceleration on modern GPUs and decent performance. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network, which can maintain the advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity simultaneously on specifically designed GPUs. Specifically, a 2 : 4 sparse network could achieve 2× speed-up without performance drop on Nvidia A100 GPUs. Furthermore, we propose a novel and effective ingredient, sparse-refined straightthrough estimator (SR-STE), to alleviate the negative influence of the approximated gradients computed by vanilla STE during optimization. We also define a metric, Sparse Architecture Divergence (SAD), to measure the sparse network's topology change during the training process. Finally, We justify SR-STE's advantages with SAD and demonstrate the effectiveness of SR-STE by performing comprehensive experiments on various tasks. Source codes and models are available at https://github.com/NM-sparsity/NM-sparsity.

show abstract

Topological Insights into Sparse Neural Networks

Cited by 3 publications

References 23 publications

SpaceNet: Make Free Space For Continual Learning

SpaceNet: Make Free Space For Continual Learning

Selfish Sparse RNN Training

Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch

Contact Info

Product

Resources

About