Bandwidth Efficient All-reduce Operation on Tree Topologies

Patarasuk, Pitch; Yuan, Xin

doi:10.1109/ipdps.2007.370405

Cited by 32 publications

(30 citation statements)

References 11 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AllReduce methods [12,33,34] also lack the flexibility to tackle straggler issues, which is more challenging than PS architecture due to the more restrictive communication pattern between workers. There are some works focusing on straggler issues in AllReduce [29,30], but their methods may lead to deadlocks and may not be able to deal with complex straggler patterns, such as transient stragglers.…”

Section: Stragglers In Distributed Model Trainingmentioning

confidence: 99%

Elastic parameter server load distribution in deep learning clusters

Chen

Peng

Bao

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

In distributed DNN training, parameter servers (PS) can become performance bottlenecks due to PS stragglers, caused by imbalanced parameter distribution, bandwidth contention, or computation interference. Few existing studies have investigated efficient parameter (aka load) distribution among PSs. We observe significant training inefficiency with the current parameter assignment in representative machine learning frameworks (e.g., MXNet, TensorFlow), and big potential for training acceleration with better PS load distribution. We design PSLD, a dynamic parameter server load distribution scheme, to mitigate PS straggler issues and accelerate distributed model training in the PS architecture. An exploitation-exploration method is carefully designed to scale in and out parameter servers and adjust parameter distribution among PSs on the go. We also design an elastic PS scaling module to carry out our scheme with little interruption to the training process. We implement our module on top of open-source PS architectures, including MXNet and BytePS. Testbed experiments show up to 2.86x speed-up in model training with PSLD, for different ML models under various straggler settings. CCS CONCEPTS • Computer systems organization → Cloud computing.

show abstract

Section: Stragglers In Distributed Model Trainingmentioning

confidence: 99%

Elastic parameter server load distribution in deep learning clusters

Chen

Peng

Bao

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

show abstract

“…For the AR architecture, there is no process dedicated just for holding variables, as shown in Figure 1(b). Rather, all workers are given a replica of variables and share locally computed gradients via collective communication primitives such as AllReduce [25,30] and AllGatherv [39]. AllReduce reduces values from all processes to a single value, while AllGatherv simply gathers the values from all processes.…”

Section: Data Parallel Training Architecturesmentioning

confidence: 99%

Parallax

Kim

Park

et al. 2019

Proceedings of the Fourteenth EuroSys Conference 2019

View full text Add to dashboard Cite

The employment of high-performance servers and GPU accelerators for training deep neural network models have greatly accelerated recent advances in deep learning (DL). DL frameworks, such as TensorFlow, MXNet, and Caffe2, have emerged to assist DL researchers to train their models in a distributed manner. Although current DL frameworks scale well for image classification models, there remain opportunities for scalable distributed training on natural language processing (NLP) models. We found that current frameworks show relatively low scalability on training NLP models due to the lack of consideration to the difference in sparsity of model parameters. In this paper, we propose Parallax, a framework that optimizes data parallel training by utilizing the sparsity of model parameters. Parallax introduces a hybrid approach that combines Parameter Server and AllReduce architectures to optimize the amount of data transfer according to the sparsity. Experiments show that Parallax built atop Tensor-Flow achieves scalable training throughput on both dense and sparse models while requiring little effort from its users. Parallax achieves up to 2.8x, 6.02x speedup for NLP models than TensorFlow and Horovod with 48 GPUs, respectively. The training speed for the image classification models is equal to Horovod and 1.53x faster than TensorFlow.

show abstract

“…AllReduce is commonly implemented with either tree [1] or butterfly [14] topologies, as shown in Figure 1a and 1b. The tree topology uses the lowest overall bandwidth, but effectively maximizes latency since the delay is set by the slowest path in the tree.…”

Section: Introductionmentioning

confidence: 99%

Butterfly Mixing: Accelerating Incremental-Update Algorithms on Clusters

Canny

Zhao

2013

Proceedings of the 2013 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

Incremental model-update strategies are widely used in machine learning and data mining. By "incremental update" we refer to models that are updated many times using small subsets of the training data. Two wellknown examples are stochastic gradient and MCMC. Both provide fast sequential performance and have generated many of the best-performing methods for particular problems (logistic regression, SVM, LDA etc.). But these methods are difficult to adapt to parallel or cluster settings because of the overhead of distributing model updates through the network. Updates can be locally batched to reduce communication overhead, but convergence typically suffers as the batch size increases. In this paper we introduce and analyze butterfly mixing, an approach which interleaves communication with computation. We evaluate butterfly mixing on stochastic gradient algorithms for logistic regression and SVM, on two datasets. Results show that butterfly mix steps are fast and failure-tolerant, and overall we achieved a 3.3x speedup over full mix (AllReduce) on an Amazon EC2 cluster.

show abstract

Bandwidth Efficient All-reduce Operation on Tree Topologies

Abstract: Abstract

Cited by 32 publications

References 11 publications

Elastic parameter server load distribution in deep learning clusters

Elastic parameter server load distribution in deep learning clusters

Parallax

Butterfly Mixing: Accelerating Incremental-Update Algorithms on Clusters

Contact Info

Product

Resources

About