On the Performance of Parallel Neural Network Implementations on Distributed Memory Architectures

Ganeshamoorthy, K.; Ranasinghe, D. N.

doi:10.1109/ccgrid.2008.68

Cited by 11 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As the discussion above, on a high-dimensional big dataset, it is essential to parallelise the training of BPNNs. In the past decades, MPI has been applied to the parallel training of BPNNs usually on supercomputers with distributed memory sharing system [7]. On the other way, GPU is also utilised to implement ANN training algorithms usually on a cluster of GPGPUs [8].…”

Section: B Existing Solutions and Remaining Challengesmentioning

confidence: 99%

See 1 more Smart Citation

Training Back Propagation Neural Networks in MapReduce on High-Dimensional Big Datasets With Global Evolution

Chen

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Owing to its scalability and high fault-tolerance even on a distributed environment built up with personal computers, MapReduce has been introduced to parallelise the training of Back Propagation Neural Networks (BPNNs) on high-dimensional big datasets. Based on the evolution of local BPNNs produced by distributed Map tasks with different data splits, the paper proposes a novel approach to the distributed data-parallel training of BPNNs in MapReduce. The approach provides a reasonable measure to get global convergent BPNN candidates from local BPNNs only convergent on the specific data splits. Further, it not only can reduce the iterations to get the global convergent BPNN, but also shows great advantages in avoiding the training to get trapped into a local optimum on high-dimensional big datasets. To improve the training efficiency further, local BPNNs from the same computing node are merged based on the average of their weight matrices before they act as individuals of the population for the global evolution. Our approach also leverages Random Project based sampling techniques to evaluate the fitness of each individual in order to lower the computation cost in the evolution stage. Experiments show that our proposed approach improves the training efficiency highly compared to the stand-alone or traditional MapReduce BPNN training, and improves model accuracy for larger datasets. The comparison with 23 other popular classification approaches also shows that our proposed approach has big advantages in accuracy.

show abstract

Section: B Existing Solutions and Remaining Challengesmentioning

confidence: 99%

“…Message Passing Interface (MPI) is widely used for the communication among processes that model a parallel program running on a distributed memory system. Experiments have shown that in a distributed memory sharing environment, BPNNs implemented with MPI have good efficiencies [7].…”

Section: Related Work a Mpi Based Parallel Learning Of Annsmentioning

confidence: 99%

Training Back Propagation Neural Networks in MapReduce on High-Dimensional Big Datasets With Global Evolution

Chen

et al. 2019

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Artificial Neural Network (ANN) merupakan komputasi yang mengambil sistem biologi yaitu jaringan saraf, jaringan saraf buatan ini berfungsi untuk melakukan komputasi klasifikasi, pengenalan pola, kontrol, forecasting, dll [1]. Algoritma ANN sebelum diuji pada sebuah lingkungan, ANN perlu melakukantraining terlebih dahulu agar algoritma ANN dapat mengenali lingkungan tersebut.…”

Section: Pendahuluanunclassified

Analisis Penggunaan Parallel Processing Multithreading Pada Resilient Backpropagation

2017

View full text Add to dashboard Cite

Abstrak-Proses pembelajaran neural network merupakan hal yang penting, bertujuan untuk mengenali lingkungan. Proses pembelajaran neural network membutuhkan waktu untuk dapat mengenali lingkungan. Terutama pada salah satu algoritma neural network yaitu resilient backpropagation. Proses untuk mempercepat pembelajaran resilient backpropagation pada penelitian ini adalah menggunakan teknik parallel processing. Teknik parallel processing yang digunakan adalah multithreading. Teknik parallel ini diterapkan pada bagian hidden layer yaitu membagi jumlah neuron pada hidden layer menjadi beberapa subproses yang dikerjakan secara bersamaan, pembagian yang dilakukan berdasarkan pada jumlah thread. Hasil yang didapatkan dalam penerapan parallel processing menggunakan teknik multithreading ke dalam algoritma resilient backpropagation membantu mempercepat waktu proses pembelajaran resilient backpropagation dengan thread yang digunakan sebanyak 3 buah thread.Keyword -Neural Network, Resilient Backpropagation, Parallel, Hybrid Partition, Multithreading. I. PENDAHULUANArtificial Neural Network (ANN) merupakan komputasi yang mengambil sistem biologi yaitu jaringan saraf, jaringan saraf buatan ini berfungsi untuk melakukan komputasi klasifikasi, pengenalan pola, kontrol, forecasting, dll [1]. Algoritma ANN sebelum diuji pada sebuah lingkungan, ANN perlu melakukantraining terlebih dahulu agar algoritma ANN dapat mengenali lingkungan tersebut. Pada saat algoritma ANN di training maka memakan waktu yang cukup lama tergantung dari besarnya dataset, nilai epoch, kecepatan pembelajaran mencapai titik konvergen [2].Proses training pada artificial neural network membutuhkan waktu untuk meyelesaikan masalah tersebut salah satu metode yang dapat digunakan adalah adaptive learning rate. Adaptivelearningrate adalah teknik yang digunakan untuk mengubah learningrate berdasarkan pada perubahan nilai errror yang dihasilkan dari setiap iterasi [3]. Selain pengunaan metode adaptive learning rate untuk mempercepat waktu training pada ANN, ada metode lain yang dapat digunakan yaitu memasukan teknik parallel kedalam arsitektur neuralnetwork tersebut. Teknik parallel adalah membagi proses menjadi subproses yang dapat dijalankan secara bersamaan.Penerapan parallel komputer terhadap backpropagation sangat membantu dalam proses training yaitu dapat memangkas waktu training [4].Resilient Propagation / Resilient Backpropagation (RPROP) adalah salah satu algoritma neuralnetwork yang dilengkapi dengan kemampuan adaptive learning yaitu kemampuan untuk menentukan learning rate yang digunakan berdasarkan nilai error gradien [5]. Algoritma RPROP dalam kasus untuk mengklasifikasikan gigitan ular menunjukan tingkat kecepatan pembelajaran untuk mencapai titik konvergen berdasarkan pada nilai batasan MSE 0.03 dimana standar backpropagation membutuhkan pembelajaran sebanyak 378 epoch sedangkan resilientbackpropagation hanya membutuhkan sebanyak 73 epoch untuk mencapai titik konvergen berdasarkan pada nilai MSE [6]. Algoritma RPROP memiliki kecepatan dalam mencapai nila...

show abstract

“…However, the accuracy loss is also a critical issue in their work. Ganeshamoorthy and Ranasinghe created a vertical partition and hybrid partition scheme [18] for parallelizing neural network using MPI (Message Passing Interface) [19]. However, MPI requires a highly homogeneous environment which decays the adaption of the parallelized algorithm.…”

Section: Related Workmentioning

confidence: 99%

Parallelizing Backpropagation Neural Network Using MapReduce and Cascading Model

Liu

Jing

2016

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Artificial Neural Network (ANN) is a widely used algorithm in pattern recognition, classification, and prediction fields. Among a number of neural networks, backpropagation neural network (BPNN) has become the most famous one due to its remarkable function approximation ability. However, a standard BPNN frequently employs a large number of sum and sigmoid calculations, which may result in low efficiency in dealing with large volume of data. Therefore to parallelize BPNN using distributed computing technologies is an effective way to improve the algorithm performance in terms of efficiency. However, traditional parallelization may lead to accuracy loss. Although several complements have been done, it is still difficult to find out a compromise between efficiency and precision. This paper presents a parallelized BPNN based on MapReduce computing model which supplies advanced features including fault tolerance, data replication, and load balancing. And also to improve the algorithm performance in terms of precision, this paper creates a cascading model based classification approach, which helps to refine the classification results. The experimental results indicate that the presented parallelized BPNN is able to offer high efficiency whilst maintaining excellent precision in enabling large-scale machine learning.

show abstract

On the Performance of Parallel Neural Network Implementations on Distributed Memory Architectures

Cited by 11 publications

References 8 publications

Training Back Propagation Neural Networks in MapReduce on High-Dimensional Big Datasets With Global Evolution

Training Back Propagation Neural Networks in MapReduce on High-Dimensional Big Datasets With Global Evolution

Analisis Penggunaan Parallel Processing Multithreading Pada Resilient Backpropagation

Parallelizing Backpropagation Neural Network Using MapReduce and Cascading Model

Contact Info

Product

Resources

About