A Survey on Auto-Parallelism of Neural Networks Training

Liang, Peng; Tao, Yu; Zhang, Xiaoda; Bai, Youhui; Su, Teng; Qiao, Linbo; Lai, Zhiquan; Li, Dongsheng

doi:10.36227/techrxiv.19522414

Cited by 2 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although it costs about 1/3 more arithmetic overheads, this method could save significant memory footprint and make it possible to train models with larger data microbatches and preserve more model parameters on each device. Activation recomputation is widely adopted [3,11,24], especially in PMP approaches, where workers might manage a large number of activations simultaneously. Merak employs and fine-tunes it to pursue higher memory efficiency.…”

Section: Systemmentioning

confidence: 99%

“…And we handle outputs of current subgraph and increase subgraph id if we create a new subgraph (lines [15][16][17][18][19][20]. After traversing all nodes, we create subgraphs with corresponding nodes, inputs and outputs (lines [23][24][25][26][27][28]. Inputs of each subgraph include all outputs of the last subgraph, thus the subgraph list could be executed as a sequence.…”

Section: Graph Sharding Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

Lai¹,

Li²,

Tang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the training process is extremely memory-intensive and communication-intensive. These features make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism and tensor model parallelism, to achieve high training efficiency.To achieve this goal, some custom software frameworks such as Megatron-LM and DeepSpeed are developed. However, current 3D parallelism frameworks still meet two issues: i) they are not transparent to model developers, which need to manually modify the model to parallelize training. ii) their utilization of computation, GPU memory and network bandwidth are not sufficient. We propose Merak, an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys with an automatic model partitioner, which uses a graph sharding algorithm on a proxy representation of the model. Merak also presents the non-intrusive API for scaling out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine in Merak. It uses several techniques to exploit available training resources, including shifted critical path pipeline schedule that brings a higher computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs show Merak can speedup the training performance over the state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42×, 1.39×, 1.43×, and 1.61×, respectively.

show abstract

Section: Systemmentioning

confidence: 99%