Performance Modeling of Distributed Deep Neural Networks

Hashemi, Seyed Mohammad; Noghabi, Shadi A.; Gropp, William; Campbell, Roy H.

doi:10.48550/arxiv.1612.00521

Cited by 2 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the best performance and alleviation overhead can come from tuning distributed algorithm and distributed system framework properties. A recent work [36] focused on analyzing DNNs performance by using CNTK framework. The performance model was to capture the scalability of the system while increasing the computation nodes in small and large clusters.…”

Section: B Experimental Evaluationmentioning

confidence: 99%

Performance Analysis and Comparison of Distributed Machine Learning Systems

Alqahtani¹,

Demirbaş²

2019

Preprint

View full text Add to dashboard Cite

Deep learning has permeated through many aspects of computing/processing systems in recent years. While distributed training architectures/frameworks are adopted for training large deep learning models quickly, there has not been a systematic study of the communication bottlenecks of these architectures and their effects on the computation cycle time and scalability. In order to analyze this problem for synchronous Stochastic Gradient Descent (SGD) training of deep learning models, we developed a performance model of computation time and communication latency under three different system architectures: Parameter Server (PS), peer-to-peer (P2P), and Ring allreduce (RA). To complement and corroborate our analytical models with quantitative results, we evaluated the computation and communication performance of these system architectures of the systems via experiments performed with Tensorflow and Horovod frameworks.We found that the system architecture has a very significant effect on the performance of training. RA-based systems achieve scalable performance as they successfully decouple network usage from the number of workers in the system. In contrast, 1PS systems suffer from low performance due to network congestion at the parameter server side. While P2P systems fare better than 1PS systems, they still suffer from significant network bottleneck. Finally, RA systems also excel by virtue of overlapping computation time and communication time, which PS and P2P architectures fail to achieve.

show abstract

Section: B Experimental Evaluationmentioning

confidence: 99%

Performance Analysis and Comparison of Distributed Machine Learning Systems

Alqahtani¹,

Demirbaş²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…CNN is more powerful and has a better performance compared to other traditional deep learning algorithms due to automatic feature extraction capability. The main drawback of CNN is the long training time and the complex neural architecture ( Hashemi et al, 2016 ; Kim et al, 2017 ). The complex neural networks structures take weeks and months to complete the training process in the case of big datasets.…”

Section: Introductionmentioning

confidence: 99%

Effect of neural network structure in accelerating performance and accuracy of a convolutional neural network with GPU/TPU for image analytics

Ravikumar

Sriraman

Saketh

et al. 2022

PeerJ Computer Science

View full text Add to dashboard Cite

Background In deep learning the most significant breakthrough in the field of image recognition, object detection language processing was done by Convolutional Neural Network (CNN). Rapid growth in data and neural networks the performance of the DNN algorithms depends on the computation power and the storage capacity of the devices. Methods In this paper, the convolutional neural network used for various image applications was studied and its acceleration in the various platforms like CPU, GPU, TPU was done. The neural network structure and the computing power and characteristics of the GPU, TPU was analyzed and summarized, the effect of these on accelerating the tasks is also explained. Cross-platform comparison of the CNN was done using three image applications the face mask detection (object detection/Computer Vision), Virus Detection in Plants (Image Classification: agriculture sector), and Pneumonia detection from X-ray Images (Image Classification/medical field). Results The CNN implementation was done and a comprehensive comparison was done on the platforms to identify the performance, throughput, bottlenecks, and training time. The CNN layer-wise execution in GPU and TPU is explained with layer-wise analysis. The impact of the fully connected layer and convolutional layer on the network is analyzed. The challenges faced during the acceleration process were discussed and future works are identified.

show abstract

Performance Modeling of Distributed Deep Neural Networks

Cited by 2 publications

References 3 publications

Performance Analysis and Comparison of Distributed Machine Learning Systems

Performance Analysis and Comparison of Distributed Machine Learning Systems

Effect of neural network structure in accelerating performance and accuracy of a convolutional neural network with GPU/TPU for image analytics

Contact Info

Product

Resources

About