Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks

Tian, Feng; Zhang, Yang; Ye, Wei; Jin, Cheng; Wu, Ziyan

doi:10.1145/3482898.3483363

Cited by 9 publications

(1 citation statement)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A distributed approach to decentralize learning in micro clouds is proposed in [14]. In [15], a method to accelerate distributed deep learning is proposed. A hybrid model with LSTM and CNN is proposed in [16] to forecast the short-term load.…”

Section: Distributed Cnn In Cloud Environmentmentioning

confidence: 99%

DConvLSTM: An Autotuning Distributed CNN Model in Cloud Environment for Optimized Accuracy, Scaling, and Execution Time

Srira

Lokesh

Sai

2022

Preprint

View full text Add to dashboard Cite

Artificial Intelligence (AI) and deep learning have altered the digital landscape. Data grows in lockstep with technological advancements. The training procedure for Deep Learning algorithms is becoming increasingly complex as the amount of data available grows. In the deep learning process, having superior technology to speed up the training and testing time is critical. CNN (Convolutional Neural Networks) is a deep learning algorithm for image processing that is widely utilized. In this paper, a novel distributed CNN with LSTM is proposed to optimize accuracy, execution time, and scalability. Different methodologies and tactics for improving CNN training time through distributed models are studied. To comprehend the neural network structure of parallelizing and spreading the execution, each layer of the CNN is investigated and analyzed. Computations on the CPU, GPU, and TPU are also investigated, as well as settings like Google Colab, AWS SageMaker, and others. According to the analysis, the use of an L2 regularization, dropout layer, and a ConvLSTM for autotuning of hyperparameters improves speed in all settings, The final CNN, with these additional layers for performance acceleration, is subsequently deployed in Google Cloud Platform Virtual Machine (VM) instances. This makes it easier to investigate the performance of the proposed distributed deep learning model at scale. Due to the communication between multiple nodes involved in distributing a deep learning model, the model's accuracy usually suffers. But in this proposed work utilizing an autotuning LSTM and Distributed CNN algorithm that distributes optimally and balances accuracy and speed-up is proposed in this proposed work. The proposed models achieved a 2.206 percent boost in speed with respect to data parallel DConvLSTM and a 1.4 percent improvement in model parallel training time of DConvLSTM. The maximum accuracy achieved was 93.3825 and 89.59 percent in data-parallel and model-parallel executions respectively.

show abstract

Section: Distributed Cnn In Cloud Environmentmentioning

confidence: 99%