Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

Liu, Shiwei; Mocanu, Decebal Constantin; Matavalam, Amarsagar Reddy Ramapuram; Pei, Yulong; Pechenizkiy, Mykola

doi:10.1007/s00521-020-05136-7

Cited by 42 publications

(45 citation statements)

References 47 publications

(54 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The efficiency of this approach has also been recently confirmed by independent researchers, who managed to train a million-node ANN on non-specialized laptops (Liu et al 2019).…”

Section: Introductionmentioning

confidence: 77%

“…A special mention goes to recent research in Liu et al (2019), where the authors managed to train a million-node ANN on non-specialized laptops, based on the SET framework that was initially introduce in Mocanu et al (2018). SET is a training procedure in which connections are pruned on the basis of their magnitude, while other connections are randomly added.…”

Section: Methods Derived From Network Science To Induce Sparse Annsmentioning

confidence: 99%

See 1 more Smart Citation

Artificial neural networks training acceleration through network science strategies

et al. 2020

View full text Add to dashboard Cite

The development of deep learning has led to a dramatic increase in the number of applications of artificial intelligence. However, the training of deeper neural networks for stable and accurate models translates into artificial neural networks (ANNs) that become unmanageable as the number of features increases. This work extends our earlier study where we explored the acceleration effects obtained by enforcing, in turn, scale freeness, small worldness, and sparsity during the ANN training process. The efficiency of that approach was confirmed by recent studies (conducted independently) where a million-node ANN was trained on non-specialized laptops. Encouraged by those results, our study is now focused on some tunable parameters, to pursue a further acceleration effect. We show that, although optimal parameter tuning is unfeasible, due to the high non-linearity of ANN problems, we can actually come up with a set of useful guidelines that lead to speed-ups in practical cases. We find that significant reductions in execution time can generally be achieved by setting the revised fraction parameter ($$\zeta $$ ζ ) to relatively low values.

show abstract

“…The efficiency of this approach has also been recently confirmed by independent researchers, who managed to train a million-node ANN on non-specialized laptops (Liu et al 2019).…”

Section: Introductionmentioning

confidence: 77%

Section: Methods Derived From Network Science To Induce Sparse Annsmentioning

confidence: 99%

Artificial neural networks training acceleration through network science strategies

et al. 2020

View full text Add to dashboard Cite

show abstract

“…We extended the sparse framework presented in Liu et al (2020b) by implementing the theoretical contributions presented in this paper. The initial implementation was sequential and it was not able to obtain the same accuracy as Keras on some datasets such as CIFAR10.…”

Section: Large Scale Sparse Neural Network Frameworkmentioning

confidence: 99%

“…In (Liu et al (2020b)), the authors developed an e icient implementation of sparse multilayer perceptrons (MLPs) trained with SET. For the first time, they built sparse MLP models with over one million artificial neurons on commodity hardware, only utilising one CPU core.…”

Section: Introductionmentioning

confidence: 99%

Truly Sparse Neural Networks at Scale

Curci

Mocanu

Pechenizkiy

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, sparse training methods have started to be established as a de facto approach for training and inference efficiency in artificial neural networks. Yet, this efficiency is just in theory. In practice, everyone uses a binary mask to simulate sparsity since the typical deep learning software and hardware are optimized for dense matrix operations. In this paper, we take an orthogonal approach, and we show that we can train truly sparse neural networks to harvest their full potential. To achieve this goal, we introduce three novel contributions, specially designed for sparse neural networks: (1) a parallel training algorithm and its corresponding sparse implementation from scratch, (2) an activation function with non-trainable parameters to favour the gradient flow, and (3) a hidden neurons importance metric to eliminate redundancies. All in one, we are able to break the record and to train the largest neural network ever trained in terms of representational power -- reaching the bat brain size. The results show that our approach has state-of-the-art performance while opening the path for an environmentally friendly artificial intelligence era.

show abstract

“…In recent years, a rapidly increasing number of research works are investigating on sparse training of DNN [112][113][114][115][116].…”

Section: 2mentioning

confidence: 99%

Deep Learning on Computational-Resource-Limited Platforms: A Survey

Chen

Zhang

et al. 2020

Mobile Information Systems

View full text Add to dashboard Cite

Nowadays, Internet of Things (IoT) gives rise to a huge amount of data. IoT nodes equipped with smart sensors can immediately extract meaningful knowledge from the data through machine learning technologies. Deep learning (DL) is constantly contributing significant progress in smart sensing due to its dramatic superiorities over traditional machine learning. The promising prospect of wide-range applications puts forwards demands on the ubiquitous deployment of DL under various contexts. As a result, performing DL on mobile or embedded platforms is becoming a common requirement. Nevertheless, a typical DL application can easily exhaust an embedded or mobile device owing to a large amount of multiply and accumulate (MAC) operations and memory access operations. Consequently, it is a challenging task to bridge the gap between deep learning and resource-limited platforms. We summarize typical applications of resource-limited deep learning and point out that deep learning is an indispensable impetus of pervasive computing. Subsequently, we explore the underlying reasons for the high computational overhead of DL through reviewing the fundamental concepts including capacity, generalization, and backpropagation of a neural network. Guided by these concepts, we investigate on principles of representative research works, as well as three types of solutions: algorithmic design, computational optimization, and hardware revolution. In pursuant to these solutions, we identify challenges to be addressed.

show abstract

Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

Cited by 42 publications

References 47 publications

Artificial neural networks training acceleration through network science strategies

Artificial neural networks training acceleration through network science strategies

Truly Sparse Neural Networks at Scale

Deep Learning on Computational-Resource-Limited Platforms: A Survey

Contact Info

Product

Resources

About