Scalable Massively Parallel Artificial Neural Networks

Abstract

There is renewed interest in computational intelligence, due to advances in algorithms, neuroscience, and computer hardware. In addition there is enormous interest in autonomous vehicles (air, ground, and sea) and robotics, which need significant onboard intelligence. Work in this area could not only lead to better understanding of the human brain but also very useful engineering applications. The functioning of the human brain is not well understood, but enormous progress has been made in understanding it and, in particular, the neocortex. There are many reasons to develop models of the brain. Artificial Neural Networks (ANN), one type of model, can be very effective for pattern recognition, function approximation, scientific classification, control, and the analysis of time series data. ANNs often use the back-propagation algorithm for training, and can require large training times especially for large networks, but there are many other types of ANNs. Once the network is trained for a particular problem, however, it can produce results in a very short time. Parallelization of ANNs could drastically reduce the training time. An object-oriented, massively-parallel ANN (Artificial Neural Network) software package SPANN (Scalable Parallel Artificial Neural Network) has been developed and is described here. MPI was used to parallelize the C++ code. Only the neurons on the edges of the domains were involved in communication, in order to reduce the communication costs and maintain scalability. The back-propagation algorithm was used to train the network. In preliminary tests, the software was used to identify character sets. The code correctly identified all the characters when adequate training was used in the network. The code was run on up to 500 Intel Itanium processors with 25,000 neurons and more than 2 billion neuron weights. Various comparisons in training time, forward propagation time, and error reduction were also made.

Biologically-Inspired Spiking Neural Networks with Hebbian Learning for Vision Processing

“…In addition, these neural networks can be trained efficiently using Hebbian-style unsupervised learning. They can also be made to be scalable on massively parallel computers [6]. Supercomputers are approaching the speed and memory of humans, but few people have regular access to more than a few hundred processors on most supercomputers.…”

“…In a previous paper [6], we showed how these 2 nd generation models could be made scalable and run efficiently on massively parallel computers. In that work, we developed an object-oriented, massively-parallel ANN software package SPANN (Scalable Parallel Artificial Neural Network).…”

Parallel Cascade Correlation Neural Network Methods for 3D Facial Recognition: A Preliminary Study

“…The two parallelization strategies on a cluster computer; training example and node parallelism using MPI approach was presented by Pethick et al [37]. Lyle et al [38] proposed the scalable massively parallel artificial neural networks for pattern recognition application. In this approach, the MPI used to parallelize the C++ code.…”

High-performance predictor for critical unstable generators based on scalable parallelized neural networks

“…Ikram et al [21] also employed cloud computing to parallelize BPNN in training phase. And also some researchers focused on solving the issue using MPI [22]. However, their ideas are all based on data separation, which does not consider the accuracy loss caused by the simple data separation.…”